Org-export-generic and wikis...

Org-export-generic and wikis...

[Robert Goldman]

I'm trying to get o-e-g to handle export to multiple wikis.

One of the ones that I'm having the most trouble with is tikiwiki
(www.tikiwiki.org).

The problem here is that tikiwiki won't reflow normal text blocks.  This
means that we need to (1) take contiguous blocks of text and ram them
all together (making sure we replace newlines with spaces) and (2) we
need to detect paragraph breaks in body text (blank lines, I believe)
and force them into the exported code as newlines.

I have a partial solution to the first, but it doesn't handle the second
problem well -- it's concatenating all the body text between two
headings together and not honoring paragraph breaks.  The reason it does
this is that the way o-e-g works is to process each line individually,
and there's no special check for blank lines.  For that matter, it
doesn't /seem/ to take into account the possibility that its output will
go somewhere that treats newlines as paragraph boundaries.

The only thing I can think about for the latter (and what I have
tentatively implemented) is to add a new export flag that indicates that
an export format expects to treat newlines as paragraph breaks.  With
this set to t, I have org-export-generic emit a newline when it sees a
blank line.  The expectation is that the value of body-line format does
/not/ emit a newline at the end.

Does this sound OK?  I will send a patch as soon as I can remember how
to make git send one in email...

Two follow-on questions about org-export-generic:
1. Would it be reasonable to move the documentation for
org-export-generic into the contrib/  directory of org-mode?  It seems
... suboptimal to have this package be maintained in the org git repo,
but its documentation in the worg git repo.  At least from my PoV this
raises the bar for keeping the documentation up-to-date and synchronized
to a pretty high level.
2. Is the existing "handle each line separately" algorithm going to
permit us to handle faces correctly?  Seems like we'll need a lot of
hair to handle, e.g., a phrase in italics that straddles a line-break,
won't we?

Best,
r

Re: OrgmodeOrg-export-generic and wikis...

[Wes Hardaker]

>>>>> On Wed, 28 Apr 2010 14:38:47 -0500, Robert Goldman <rpgoldman@sift.info> said:

RG> 1. Would it be reasonable to move the documentation for
RG> org-export-generic into the contrib/ directory of org-mode?  It
RG> seems ... suboptimal to have this package be maintained in the org
RG> git repo, but its documentation in the worg git repo.  At least from
RG> my PoV this raises the bar for keeping the documentation up-to-date
RG> and synchronized to a pretty high level.

I'll let Carsten answer questions like that ;-)

RG> 2. Is the existing "handle each line separately" algorithm going to
RG> permit us to handle faces correctly?  Seems like we'll need a lot of
RG> hair to handle, e.g., a phrase in italics that straddles a line-break,
RG> won't we?

Well, that's a good question and one I don't have a great answer for.
We could switch the parser to be multi-line based regexp matches but I
suspect the complexity of when to stop those regexps will get rather
harry too!
-- 
Wes Hardaker                                     
My Pictures:  http://capturedonearth.com/
My Thoughts:  http://pontifications.hardakers.net/

Re: Re: OrgmodeOrg-export-generic and wikis...

[Carsten Dominik]

On Apr 29, 2010, at 3:57 PM, Wes Hardaker wrote:

>>>>>> On Wed, 28 Apr 2010 14:38:47 -0500, Robert Goldman <rpgoldman@sift.info 
>>>>>> > said:
>
> RG> 1. Would it be reasonable to move the documentation for
> RG> org-export-generic into the contrib/ directory of org-mode?  It
> RG> seems ... suboptimal to have this package be maintained in the org
> RG> git repo, but its documentation in the worg git repo.  At least  
> from
> RG> my PoV this raises the bar for keeping the documentation up-to- 
> date
> RG> and synchronized to a pretty high level.
>
> I'll let Carsten answer questions like that ;-)

The reason for this is the following.

Many of the contributed packages where written by people who were active
in Org-mode for a while and then less so.  Many of these package had no
documentation at all.  So I started a page on WOrg where this
documentation can be added and edited by other users, with quite
some success - now most package do have documentation.

Keeping documentation for a contributed package the the org-repo would  
be OK, but there would be no mechanism to automatically put the latest  
version up on on the web.

Changing this would require a volunteer who commits to keep the  
documentation
of contributed packages in a consistent and web-publishable way in the  
contrib directory.

- Carsten

>
> RG> 2. Is the existing "handle each line separately" algorithm going  
> to
> RG> permit us to handle faces correctly?  Seems like we'll need a  
> lot of
> RG> hair to handle, e.g., a phrase in italics that straddles a line- 
> break,
> RG> won't we?
>
> Well, that's a good question and one I don't have a great answer for.
> We could switch the parser to be multi-line based regexp matches but I
> suspect the complexity of when to stop those regexps will get rather
> harry too!
> -- 
> Wes Hardaker
> My Pictures:  http://capturedonearth.com/
> My Thoughts:  http://pontifications.hardakers.net/
>
>
> _______________________________________________
> Emacs-orgmode mailing list
> Please use `Reply All' to send replies to the list.
> Emacs-orgmode@gnu.org
> http://lists.gnu.org/mailman/listinfo/emacs-orgmode

- Carsten

Re: Re: OrgmodeOrg-export-generic and wikis...

[Robert Goldman]

On 4/29/10 Apr 29 -9:14 AM, Carsten Dominik wrote:
> 
> On Apr 29, 2010, at 3:57 PM, Wes Hardaker wrote:
> 
>>>>>>> On Wed, 28 Apr 2010 14:38:47 -0500, Robert Goldman
>>>>>>> <rpgoldman@sift.info> said:
>>
>> RG> 1. Would it be reasonable to move the documentation for
>> RG> org-export-generic into the contrib/ directory of org-mode?  It
>> RG> seems ... suboptimal to have this package be maintained in the org
>> RG> git repo, but its documentation in the worg git repo.  At least from
>> RG> my PoV this raises the bar for keeping the documentation up-to-date
>> RG> and synchronized to a pretty high level.
>>
>> I'll let Carsten answer questions like that ;-)
> 
> The reason for this is the following.
> 
> Many of the contributed packages where written by people who were active
> in Org-mode for a while and then less so.  Many of these package had no
> documentation at all.  So I started a page on WOrg where this
> documentation can be added and edited by other users, with quite
> some success - now most package do have documentation.
> 
> Keeping documentation for a contributed package the the org-repo would
> be OK, but there would be no mechanism to automatically put the latest
> version up on on the web.
> 
> Changing this would require a volunteer who commits to keep the
> documentation
> of contributed packages in a consistent and web-publishable way in the
> contrib directory.

Here's the particular problem for org-export-generic --- maybe there's a
solution:

org-export-generic is primarily data-driven.  To specify an export
technique, you populate a very big data structure using a macro with
what look like common-lisp keywords.  Here's an example:



(org-set-generic-type
 "tikiwiki"
 '(:file-suffix     ".txt"
   :key-binding     ?U

   ;; lifted from wikipedia exporter
   :header-prefix            	    ""
   :header-suffix            	    ""

   :title-format             	    "-= %s =-\n"

   :date-export        	    nil

   :toc-export                    nil

   :body-header-section-numbers   nil
   :body-section-prefix           "\n"

   :body-section-header-prefix    ("! " "!! " "!!! " "!!!! "
                                   "!!!!! " "!!!!!! " "!!!!!!! ")
   :body-section-header-suffix    (" \n" " \n" " \n"
                                   " \n" " \n" " \n")


   :body-line-export-preformated  t ;; yes/no/maybe???
   :body-line-format              "%s "
   :body-line-wrap                nil

   :body-line-fixed-format       " %s\n"

   :body-list-format              "* %s\n"
   :body-number-list-format       "# %s\n"
;;    :body-list-prefix              "LISTSTART"
;;    :body-list-suffix              "LISTEND"
   :blockquote-start              "\n^\n"
   :blockquote-end                "^\n\n"
   :body-newline-paragraph        t
   ))

The problem is that this is VERY difficult to document as the set of
keywords expands (e.g., I add :body-newline-paragraph, :blockquote-start
and :blockquote-end).  These aren't arguments, so they can't get
documented in the code in a docstring.  org-set-generic-type is a
function, not a mode, so there's no docstring for the mode to hold the
documentation.

This is already not working, AFAICT, the worg docs don't seem to be
complete or accurate.

I'm pretty convinced from general code and document-writing practice
that the best solution would be one that puts the documentation as close
to the code as possible.

If this were common-lisp, I would add a new method to the DOCUMENTATION
generic function, so that one could say

(documentation :body-newline-paragraph :org-export-keyword)

Then we could add a declaration macro, and put the docstring there:

(def-generic-export-keyword :body-newline-paragraph
  :boolean
  "Should newlines ONLY be used as paragraph breaks.  If
the associated value is true, then org-export-generic will
flow contiguous paragraphs into one long line, adding newlines
only where there is a blank line.  Should be coupled with a
value for :body-line-format that does NOT contain a newline
character, e.g., \"%s \"")

I suppose we could add something like this, and possibly even
write a script that would blat the docstring into something that
Worg could display.

Any thoughts?

Best,

r

Re: Re: OrgmodeOrg-export-generic and wikis...

[Carsten Dominik]

Hi Robert, Wes,

I am not sure if there is any action  should be taking here?
I have no problems bundling some documentation with the distribution -  
let me just know what you want and send me what I should distribute.

- Carsten

On Apr 29, 2010, at 5:18 PM, Robert Goldman wrote:

> On 4/29/10 Apr 29 -9:14 AM, Carsten Dominik wrote:
>>
>> On Apr 29, 2010, at 3:57 PM, Wes Hardaker wrote:
>>
>>>>>>>> On Wed, 28 Apr 2010 14:38:47 -0500, Robert Goldman
>>>>>>>> <rpgoldman@sift.info> said:
>>>
>>> RG> 1. Would it be reasonable to move the documentation for
>>> RG> org-export-generic into the contrib/ directory of org-mode?  It
>>> RG> seems ... suboptimal to have this package be maintained in the  
>>> org
>>> RG> git repo, but its documentation in the worg git repo.  At  
>>> least from
>>> RG> my PoV this raises the bar for keeping the documentation up-to- 
>>> date
>>> RG> and synchronized to a pretty high level.
>>>
>>> I'll let Carsten answer questions like that ;-)
>>
>> The reason for this is the following.
>>
>> Many of the contributed packages where written by people who were  
>> active
>> in Org-mode for a while and then less so.  Many of these package  
>> had no
>> documentation at all.  So I started a page on WOrg where this
>> documentation can be added and edited by other users, with quite
>> some success - now most package do have documentation.
>>
>> Keeping documentation for a contributed package the the org-repo  
>> would
>> be OK, but there would be no mechanism to automatically put the  
>> latest
>> version up on on the web.
>>
>> Changing this would require a volunteer who commits to keep the
>> documentation
>> of contributed packages in a consistent and web-publishable way in  
>> the
>> contrib directory.
>
> Here's the particular problem for org-export-generic --- maybe  
> there's a
> solution:
>
> org-export-generic is primarily data-driven.  To specify an export
> technique, you populate a very big data structure using a macro with
> what look like common-lisp keywords.  Here's an example:
>
>
>
> (org-set-generic-type
> "tikiwiki"
> '(:file-suffix     ".txt"
>   :key-binding     ?U
>
>   ;; lifted from wikipedia exporter
>   :header-prefix            	    ""
>   :header-suffix            	    ""
>
>   :title-format             	    "-= %s =-\n"
>
>   :date-export        	    nil
>
>   :toc-export                    nil
>
>   :body-header-section-numbers   nil
>   :body-section-prefix           "\n"
>
>   :body-section-header-prefix    ("! " "!! " "!!! " "!!!! "
>                                   "!!!!! " "!!!!!! " "!!!!!!! ")
>   :body-section-header-suffix    (" \n" " \n" " \n"
>                                   " \n" " \n" " \n")
>
>
>   :body-line-export-preformated  t ;; yes/no/maybe???
>   :body-line-format              "%s "
>   :body-line-wrap                nil
>
>   :body-line-fixed-format       " %s\n"
>
>   :body-list-format              "* %s\n"
>   :body-number-list-format       "# %s\n"
> ;;    :body-list-prefix              "LISTSTART"
> ;;    :body-list-suffix              "LISTEND"
>   :blockquote-start              "\n^\n"
>   :blockquote-end                "^\n\n"
>   :body-newline-paragraph        t
>   ))
>
> The problem is that this is VERY difficult to document as the set of
> keywords expands (e.g., I add :body-newline-paragraph, :blockquote- 
> start
> and :blockquote-end).  These aren't arguments, so they can't get
> documented in the code in a docstring.  org-set-generic-type is a
> function, not a mode, so there's no docstring for the mode to hold the
> documentation.
>
> This is already not working, AFAICT, the worg docs don't seem to be
> complete or accurate.
>
> I'm pretty convinced from general code and document-writing practice
> that the best solution would be one that puts the documentation as  
> close
> to the code as possible.
>
> If this were common-lisp, I would add a new method to the  
> DOCUMENTATION
> generic function, so that one could say
>
> (documentation :body-newline-paragraph :org-export-keyword)
>
> Then we could add a declaration macro, and put the docstring there:
>
> (def-generic-export-keyword :body-newline-paragraph
>  :boolean
>  "Should newlines ONLY be used as paragraph breaks.  If
> the associated value is true, then org-export-generic will
> flow contiguous paragraphs into one long line, adding newlines
> only where there is a blank line.  Should be coupled with a
> value for :body-line-format that does NOT contain a newline
> character, e.g., \"%s \"")
>
> I suppose we could add something like this, and possibly even
> write a script that would blat the docstring into something that
> Worg could display.
>
> Any thoughts?
>
> Best,
>
> r
>
>

- Carsten

Re: Re: OrgmodeOrg-export-generic and wikis...

[Robert Goldman]

On 5/14/10 May 14 -2:16 AM, Carsten Dominik wrote:
> Hi Robert, Wes,
> 
> I am not sure if there is any action  should be taking here?
> I have no problems bundling some documentation with the distribution -
> let me just know what you want and send me what I should distribute.

I have sent Wes a couple of proposed patches for inspection.  We should
probably wait until he's had a chance to look at them.  They include the
introduction of a declaration macro for the keywords that are used to
configure a generic exporter.

Also, I should probably get the fontification fully done before pushing.

Question:  do you ever have a shared branch?  If it would be more
convenient for you and Wes, I suppose I could be pushing my patches onto
a branch on your repo, instead of emailing patches.  Whatever works best
for you two.

Cheers,
r

> 
> - Carsten
> 
> On Apr 29, 2010, at 5:18 PM, Robert Goldman wrote:
> 
>> On 4/29/10 Apr 29 -9:14 AM, Carsten Dominik wrote:
>>>
>>> On Apr 29, 2010, at 3:57 PM, Wes Hardaker wrote:
>>>
>>>>>>>>> On Wed, 28 Apr 2010 14:38:47 -0500, Robert Goldman
>>>>>>>>> <rpgoldman@sift.info> said:
>>>>
>>>> RG> 1. Would it be reasonable to move the documentation for
>>>> RG> org-export-generic into the contrib/ directory of org-mode?  It
>>>> RG> seems ... suboptimal to have this package be maintained in the org
>>>> RG> git repo, but its documentation in the worg git repo.  At least
>>>> from
>>>> RG> my PoV this raises the bar for keeping the documentation up-to-date
>>>> RG> and synchronized to a pretty high level.
>>>>
>>>> I'll let Carsten answer questions like that ;-)
>>>
>>> The reason for this is the following.
>>>
>>> Many of the contributed packages where written by people who were active
>>> in Org-mode for a while and then less so.  Many of these package had no
>>> documentation at all.  So I started a page on WOrg where this
>>> documentation can be added and edited by other users, with quite
>>> some success - now most package do have documentation.
>>>
>>> Keeping documentation for a contributed package the the org-repo would
>>> be OK, but there would be no mechanism to automatically put the latest
>>> version up on on the web.
>>>
>>> Changing this would require a volunteer who commits to keep the
>>> documentation
>>> of contributed packages in a consistent and web-publishable way in the
>>> contrib directory.
>>
>> Here's the particular problem for org-export-generic --- maybe there's a
>> solution:
>>
>> org-export-generic is primarily data-driven.  To specify an export
>> technique, you populate a very big data structure using a macro with
>> what look like common-lisp keywords.  Here's an example:
>>
>>
>>
>> (org-set-generic-type
>> "tikiwiki"
>> '(:file-suffix     ".txt"
>>   :key-binding     ?U
>>
>>   ;; lifted from wikipedia exporter
>>   :header-prefix                    ""
>>   :header-suffix                    ""
>>
>>   :title-format                     "-= %s =-\n"
>>
>>   :date-export                nil
>>
>>   :toc-export                    nil
>>
>>   :body-header-section-numbers   nil
>>   :body-section-prefix           "\n"
>>
>>   :body-section-header-prefix    ("! " "!! " "!!! " "!!!! "
>>                                   "!!!!! " "!!!!!! " "!!!!!!! ")
>>   :body-section-header-suffix    (" \n" " \n" " \n"
>>                                   " \n" " \n" " \n")
>>
>>
>>   :body-line-export-preformated  t ;; yes/no/maybe???
>>   :body-line-format              "%s "
>>   :body-line-wrap                nil
>>
>>   :body-line-fixed-format       " %s\n"
>>
>>   :body-list-format              "* %s\n"
>>   :body-number-list-format       "# %s\n"
>> ;;    :body-list-prefix              "LISTSTART"
>> ;;    :body-list-suffix              "LISTEND"
>>   :blockquote-start              "\n^\n"
>>   :blockquote-end                "^\n\n"
>>   :body-newline-paragraph        t
>>   ))
>>
>> The problem is that this is VERY difficult to document as the set of
>> keywords expands (e.g., I add :body-newline-paragraph, :blockquote-start
>> and :blockquote-end).  These aren't arguments, so they can't get
>> documented in the code in a docstring.  org-set-generic-type is a
>> function, not a mode, so there's no docstring for the mode to hold the
>> documentation.
>>
>> This is already not working, AFAICT, the worg docs don't seem to be
>> complete or accurate.
>>
>> I'm pretty convinced from general code and document-writing practice
>> that the best solution would be one that puts the documentation as close
>> to the code as possible.
>>
>> If this were common-lisp, I would add a new method to the DOCUMENTATION
>> generic function, so that one could say
>>
>> (documentation :body-newline-paragraph :org-export-keyword)
>>
>> Then we could add a declaration macro, and put the docstring there:
>>
>> (def-generic-export-keyword :body-newline-paragraph
>>  :boolean
>>  "Should newlines ONLY be used as paragraph breaks.  If
>> the associated value is true, then org-export-generic will
>> flow contiguous paragraphs into one long line, adding newlines
>> only where there is a blank line.  Should be coupled with a
>> value for :body-line-format that does NOT contain a newline
>> character, e.g., \"%s \"")
>>
>> I suppose we could add something like this, and possibly even
>> write a script that would blat the docstring into something that
>> Worg could display.
>>
>> Any thoughts?
>>
>> Best,
>>
>> r
>>
>>
> 
> - Carsten
> 
> 
>