[BUG] [ODT] Annotations break paragraphs

[BUG] [ODT] Annotations break paragraphs

[Christian Moe]

Hi,

Paragraphs currently break around ODT annotations when they
shouldn't. Annotations are a useful feature of the ODT exporter:

  There is an annotation by the original author here
  #+BEGIN_ANNOTATION
    I never meant to break this paragraph.
  #+END_ANNOTATION
  in the middle of the paragraph.

The expected result would be one paragraph, with the annotation
displayed as a comment in the page margin, anchored after the word
"here".

The new exporter adds paragraph breaks around the anchor for the
marginal comment. This is the wrong behavior in all cases. These
comments are meant to be anchored inside paragraphs that are not meant
to be broken. (Using a fresh Org-mode version 8.0-pre on Emacs 24.3.1.)

The old exporter simply wrapped the contents of the annotation block
(plus date/author info) in
"<office:annotation>\n%s\n</office:annotation>" within the
paragraph. 

The ODT exporter's annotation blocks are a bit of a special case. They
use special-block syntax. But they're not really blocks, since they're
supposed to live inside other blocks (paragraphs). I suppose that doesn't
agree well with the new exporter.

Yours,
Christian

Re: [BUG] [ODT] Annotations break paragraphs

[Samuel Wales]

Hi Christian,

On 3/21/13, Christian Moe <mail@christianmoe.com> wrote:
> The new exporter adds paragraph breaks around the anchor for the
> marginal comment. This is the wrong behavior in all cases. These
> comments are meant to be anchored inside paragraphs that are not meant
> to be broken. (Using a fresh Org-mode version 8.0-pre on Emacs 24.3.1.)

A similar issue arises with inline footnotes.

Over the years I have built a lot of documents, each a
subtree, that use them.  To me, this naturally and
definitely looks like a single paragraph with a 2-paragraph
footnote, regardless of what the new exporter thinks:

  a[fn::b

  c] d e.

In my case I was able with Nicolas's supplied code to create a hook
that normalized footnotes before export.  Maybe extracting in a hook
will work for you.  However, I fear that incorporating the parser into
the font lock engine will break these paragraphs.  :(

Samuel

-- 
The Kafka Pandemic: http://thekafkapandemic.blogspot.com

The disease DOES progress.  MANY people have died from it.  ANYBODY
can get it.  There is NO hope without action.  This means YOU.

Re: [BUG] [ODT] Annotations break paragraphs

[Christian Moe]

Samuel Wales writes:
>
> A similar issue arises with inline footnotes.
[...]
> In my case I was able with Nicolas's supplied code to create a hook
> that normalized footnotes before export.  Maybe extracting in a hook
> will work for you.  However, I fear that incorporating the parser into
> the font lock engine will break these paragraphs.

Thanks for the tip. I don't think user-side hacks are the way to go
here, though. Org-odt provides an annotation feature for ODT export,
based on using the special-block syntax, that no longer works as
intended. I'm hoping it can simply be fixed, but if putting the
annotation in a block must lead to paragraph breaks under the new
exporter, a different solution is needed.

Yours,
Christian

Re: [BUG] [ODT] Annotations break paragraphs

[Achim Gratz]

Am 25.03.2013 11:29, schrieb Christian Moe:
> Thanks for the tip. I don't think user-side hacks are the way to go
> here, though. Org-odt provides an annotation feature for ODT export,
> based on using the special-block syntax, that no longer works as
> intended. I'm hoping it can simply be fixed, but if putting the
> annotation in a block must lead to paragraph breaks under the new
> exporter, a different solution is needed.

I've been trying to sell Nicolas the idea of paragraph continuations, 
but so far I haven't had much luck.  It's OK (I think) that Org intially 
parses this as two paragraphs with an annotation inbetween, but there 
should be a way for the exporter to pull the two paragraphs together 
based on context (blank lines or not, etc.).


Regards,
-- 
Achim.

(on the road :-)

Re: [BUG] [ODT] Annotations break paragraphs

[Bastien]

Hi Christian,

Christian Moe <mail@christianmoe.com> writes:

> Paragraphs currently break around ODT annotations when they
> shouldn't. Annotations are a useful feature of the ODT exporter:
>
>   There is an annotation by the original author here
>   #+BEGIN_ANNOTATION
>     I never meant to break this paragraph.
>   #+END_ANNOTATION
>   in the middle of the paragraph.
>
> The expected result would be one paragraph, with the annotation
> displayed as a comment in the page margin, anchored after the word
> "here".
>
> The new exporter adds paragraph breaks around the anchor for the
> marginal comment. 

Fixed, thanks.

-- 
 Bastien

Re: [BUG] [ODT] Annotations break paragraphs

[Christian Moe]

Hi, Bastien,

Thanks for looking into this. I just pulled and tested, but I cannot
confirm the fix yet. I still get paragraph breaks around annotations
with Org-mode version 8.0-pre (release_8.0-pre-219-g8eb0d6).

Yours,
Christian


Bastien writes:

> Hi Christian,
>
> Christian Moe <mail@christianmoe.com> writes:
>
>> Paragraphs currently break around ODT annotations when they
>> shouldn't. Annotations are a useful feature of the ODT exporter:
>>
>>   There is an annotation by the original author here
>>   #+BEGIN_ANNOTATION
>>     I never meant to break this paragraph.
>>   #+END_ANNOTATION
>>   in the middle of the paragraph.
>>
>> The expected result would be one paragraph, with the annotation
>> displayed as a comment in the page margin, anchored after the word
>> "here".
>>
>> The new exporter adds paragraph breaks around the anchor for the
>> marginal comment. 
>
> Fixed, thanks.

Re: [BUG] [ODT] Annotations break paragraphs

[Bastien]

Hi Christian,

Christian Moe <mail@christianmoe.com> writes:

> Thanks for looking into this. I just pulled and tested, but I cannot
> confirm the fix yet. I still get paragraph breaks around annotations
> with Org-mode version 8.0-pre (release_8.0-pre-219-g8eb0d6).

Before the fix, I could see no annotation at all in the ODT export;
after the fix, I see the annotation -- do you confirm *this* problem 
at least existed for you, and does not exist anymore?  

As for annotation breaking the paragraph, I'll digg further.

Best,

-- 
 Bastien

Re: [BUG] [ODT] Annotations break paragraphs

[Achim Gratz]

Am 25.03.2013 17:12, schrieb Christian Moe:
> Thanks for looking into this. I just pulled and tested, but I cannot
> confirm the fix yet. I still get paragraph breaks around annotations
> with Org-mode version 8.0-pre (release_8.0-pre-219-g8eb0d6).

It can't be fixed this way since annotations end the paragraph and 
whatever comes next is a new element.  The ODT exporter gets two 
paragraphs and has no way of knowing that these should actually be 
exported as a single paragraph.  Inline source blocks might work (i.e. 
src_ANNOTATION{...}), I don't know.


-- 
Achim.

(on the road :-)

Re: [BUG] [ODT] Annotations break paragraphs

[Christian Moe]

Achim Gratz writes:
> It can't be fixed this way since annotations end the paragraph and 
> whatever comes next is a new element.  The ODT exporter gets two 
> paragraphs and has no way of knowing that these should actually be 
> exported as a single paragraph.

Yeah, that's what I was afraid of.

>  Inline source blocks might work (i.e. 
> src_ANNOTATION{...}), I don't know.

Sure. I had a macro solution going before I discovered Jambunathan had
put in an annotation feature. I've updated it now to use export snippets:

#+MACRO: comment @@odt:<office:annotation><dc:creator>@@{{{author}}}@@odt:</dc:creator><dc:date>@@{{{date(%Y-%m-%dT%T)}}}@@odt:</dc:date><text:p text:style-name="P1"><text:span text:style-name="T1">$1</text:span></text:p></office:annotation>@@


That allows using annotations like this{{{comment(You can annotate with
macros\, but remember to escape your commas)}}}.

Yours,
Christian

Re: [BUG] [ODT] Annotations break paragraphs

[Bastien]

Hi,

please have a go against latest HEAD and let me know if it works.  I
tried with annotations at the beginning of a section, of a paragraph,
in the middle of a paragraph, at the end of a paragraph and at the end
of a section.  The fix qualifies as the Most Ugly Hack On Earth, but
does the job for me.

Thanks,

-- 
 Bastien

Re: [BUG] [ODT] Annotations break paragraphs

[Christian Moe]

Hi,

No, sorry. I see the same issues as in my previous message: an
annotation at the end of a paragraph swallows the (intended) paragraph
break before the next paragraph; an annotation before a list item causes
a format error. 

The last commit I see from you is at 18:28:50, though: "Fix previous
commit again / Now off." Should I be looking at something newer?

Yours,
Christian

Bastien writes:

> Hi,
>
> please have a go against latest HEAD and let me know if it works.  I
> tried with annotations at the beginning of a section, of a paragraph,
> in the middle of a paragraph, at the end of a paragraph and at the end
> of a section.  The fix qualifies as the Most Ugly Hack On Earth, but
> does the job for me.
>
> Thanks,

Re: [BUG] [ODT] Annotations break paragraphs

[Bastien]

Hi Christian,

Christian Moe <mail@christianmoe.com> writes:

> Thanks for looking into this. I just pulled and tested, but I cannot
> confirm the fix yet. I still get paragraph breaks around annotations
> with Org-mode version 8.0-pre (release_8.0-pre-219-g8eb0d6).

I gave it another try.  Please let me know.

-- 
 Bastien

Re: [BUG] [ODT] Annotations break paragraphs

[Achim Gratz]

Am 25.03.2013 18:05, schrieb Bastien:
> I gave it another try.  Please let me know.

Now add an annotation at the end of a paragraph... it simply doesn't 
work unless org-element gets proper support for telling the exporter 
which Org "paragraph" elements should be exported together as a single 
paragraph.


Regards,
-- 
Achim.

(on the road :-)

Re: [BUG] [ODT] Annotations break paragraphs

[Nicolas Goaziou]

Hello,

Achim Gratz <Stromeko@Nexgo.DE> writes:

> Am 25.03.2013 18:05, schrieb Bastien:
>> I gave it another try.  Please let me know.
>
> Now add an annotation at the end of a paragraph... it simply doesn't
> work unless org-element gets proper support for telling the exporter
> which Org "paragraph" elements should be exported together as a single
> paragraph.

Each export back-end has full control over the parse tree generated by
Elements, and each transcoding function has full access to the context
of the element being transcoded.

There are a few examples of this already in back-ends. Hence, ox-latex
will wrap contiguous tables sharing the same math mode within the same
environment.

I didn't look at the problem (nor at Bastien's solution): could someone
post the proper code that should be generated?


Regards,

-- 
Nicolas Goaziou

Re: [BUG] [ODT] Annotations break paragraphs

[Christian Moe]

Nicolas Goaziou writes:
> I didn't look at the problem (nor at Bastien's solution): could someone
> post the proper code that should be generated?

Hi,

I'll try.

This Org code:

   A paragraph.
   #+begin_annotation
     An annotation.
   #+end_annotation

   Another paragraph.


...should result in this ODT XML snippet:


<text:p text:style-name="Text_20_body">A paragraph.<office:annotation><dc:creator>Christian Moe</dc:creator><text:p text:style-name="Text_20_body">An annotation.</text:p></office:annotation></text:p>
<text:p>Another paragraph.</text:p>


It should not close the first paragraph before the <office:annotation>
element, and it should not wrap that element in a paragraph of its own,
as it did before this evening's fixes.

But it should close the first paragraph with </text:p> after the
<office:annotation> element, and it should open a new <text:p> element
for the second paragraph. The fixes I tested did not.

Hope this made sense.

Yours,
Christian

Re: [BUG] [ODT] Annotations break paragraphs

[Christian Moe]

Hi, Bastien,

Thanks. Annotations now work inside paragraphs, where they now leave
only an extra space instead of paragraph breaks.

However, an annotation at the end of a paragraph swallows up the
*intended* paragraph break before the next paragraph.

And the fix doesn't seem to be quite safe. An annotation immediately
before a list item causes an ODT document error -- invalid XML.

Examples below.

Yours,
Christian

* Test 1

A paragraph.
#+begin_annotation
  An annotation at the end of a paragraph
#+end_annotation

Another paragraph that should not be run together with the first, but
is.


* Test 2

1. A list item.

2. A list item with an annotation at the end.
   #+begin_annotation
   This is an annotation.
   #+end_annotation

3. Another list item. A format error is triggered.



Bastien writes:

> Hi Christian,
>
> Christian Moe <mail@christianmoe.com> writes:
>
>> Thanks for looking into this. I just pulled and tested, but I cannot
>> confirm the fix yet. I still get paragraph breaks around annotations
>> with Org-mode version 8.0-pre (release_8.0-pre-219-g8eb0d6).
>
> I gave it another try.  Please let me know.

Re: [BUG] [ODT] Annotations break paragraphs

[Bastien]

Hi Christian,

okay, I reverted my wrong fixes.  I'll let Nicolas have a look.

I would not favor a solution that allows more #+begin_ blocks to
be inlined.

The proper way to handle this is to introduce a new syntax for
inlined annotations and to treat them appropriately in exporters.

Since we have both #+begin_src and src_<lang>{...} I'd suggest
having annotation_{...} or something similar.

The LaTeX exporter could use \marginpar{...} and the HTML back-end
could make them appear when hovering with the mouse on the annotated
parts (just an idea.)

Maybe we will have to live with the current "regression" for 8.0
and implement the new syntax for 8.1.  Or for 8.0, if Nicolas thinks
the change is okay and not too error prone.

Best,

-- 
 Bastien

Re: [BUG] [ODT] Annotations break paragraphs

[Nicolas Goaziou]

[-- Attachment #1: Type: text/plain, Size: 1165 bytes --]

Hello,

Bastien <bzg@altern.org> writes:

> okay, I reverted my wrong fixes.  I'll let Nicolas have a look.

I attach a patch (hardly tested) for that. Does it work as expected?

> I would not favor a solution that allows more #+begin_ blocks to
> be inlined.

Neither would I. Blocks are containers.

> The proper way to handle this is to introduce a new syntax for
> inlined annotations and to treat them appropriately in exporters.
>
> Since we have both #+begin_src and src_<lang>{...} I'd suggest
> having annotation_{...} or something similar.

I would suggest [annotation:label] or [note:label] a dedicated section
for contents, much like footnotes (aren't they just special footnotes,
after all?). That way, they can be inlined while still being able to
contain paragraphs.

> The LaTeX exporter could use \marginpar{...} and the HTML back-end
> could make them appear when hovering with the mouse on the annotated
> parts (just an idea.)
>
> Maybe we will have to live with the current "regression" for 8.0
> and implement the new syntax for 8.1.  Or for 8.0, if Nicolas thinks
> the change is okay and not too error prone.


Regards,

-- 
Nicolas Goaziou

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: fix annotate blocks --]
[-- Type: text/x-patch, Size: 3855 bytes --]

From 502e29c49a1d95eb853550f157567ffc328403c6 Mon Sep 17 00:00:00 2001
From: Nicolas Goaziou <n.goaziou@gmail.com>
Date: Mon, 25 Mar 2013 23:07:04 +0100
Subject: [PATCH] ox-odt: Properly handle ANNOTATE special blocks

* lisp/ox-odt.el (org-odt-paragraph, org-odt-special-block): Properly
  handle ANNOTATE special blocks associated to paragraphs.
---
 lisp/ox-odt.el | 59 +++++++++++++++++++++++++++++++++++++++++++++++-----------
 1 file changed, 48 insertions(+), 11 deletions(-)

diff --git a/lisp/ox-odt.el b/lisp/ox-odt.el
index 9dd8946..5896d52 100644
--- a/lisp/ox-odt.el
+++ b/lisp/ox-odt.el
@@ -2888,10 +2888,29 @@ the plist used as a communication channel."
     ;; item has a checkbox, splice the checkbox and paragraph contents
     ;; together.
     (when (and (eq (org-element-type parent) 'item)
-    	       (eq paragraph (car (org-element-contents parent))))
+	       (eq paragraph (car (org-element-contents parent))))
       (setq contents (concat (org-odt--checkbox parent) contents)))
     (assert style)
-    (format "\n<text:p text:style-name=\"%s\">%s</text:p>" style contents)))
+    (concat
+     ;; Open paragraph, unless it is preceded by an annotation with no
+     ;; blank line at its end.
+     (unless (let ((prev (org-export-get-previous-element paragraph info)))
+	       (and (eq (org-element-type prev) 'special-block)
+		    (equal (org-element-property :type prev) "ANNOTATION")
+		    (let ((blank (org-element-property :post-blank prev)))
+		      (or (not blank) (zerop blank)))))
+       (format "<text:p text:style-name=\"%s\">" style))
+     ;; Paragraph's contents.
+     contents
+     ;; Close paragraph, unless it is followed by an annotation with
+     ;; no blank line in-between.
+     (unless (and (let ((blank (org-element-property :post-blank paragraph)))
+		    (or (not blank) (zerop blank)))
+		  (let ((next (org-export-get-next-element paragraph info)))
+		    (and (eq (org-element-type next) 'special-block)
+			 (equal (org-element-property :type next)
+				"ANNOTATION"))))
+       "</text:p>"))))
 
 
 ;;;; Plain List
@@ -3066,15 +3085,33 @@ holding contextual information."
 	     (date (or (plist-get attributes :date)
 		       ;; FIXME: Is `car' right thing to do below?
 		       (car (plist-get info :date)))))
-	(format "\n<text:p>%s</text:p>"
-		(format "<office:annotation>\n%s\n</office:annotation>"
-			(concat
-			 (and author
-			      (format "<dc:creator>%s</dc:creator>" author))
-			 (and date
-			      (format "<dc:date>%s</dc:date>"
-				      (org-odt--format-timestamp date nil 'iso-date)))
-			 contents)))))
+	(concat
+	 ;; Annotation starts a paragraph when there's no paragraph
+	 ;; before it or the paragraph before ends with some blank
+	 ;; lines.  In that case, start a new paragraph.
+	 (let ((prev (org-export-get-previous-element special-block info)))
+	   (unless (and (eq (org-element-type prev) 'paragraph)
+			(let ((blank (org-element-property :post-blank prev)))
+			  (or (not blank) (zerop blank))))
+	     "<text:p text:style-name=\"Text_20_body\">"))
+	 (format "<office:annotation>\n%s\n</office:annotation>"
+		 (concat
+		  (and author
+		       (format "<dc:creator>%s</dc:creator>" author))
+		  (and date
+		       (format "<dc:date>%s</dc:date>"
+			       (org-odt--format-timestamp date nil 'iso-date)))
+		  contents))
+	 ;; Annotation ends a paragraph when it ends with blank lines
+	 ;; or when no paragraph follows it.  In that case, end
+	 ;; current paragraph.
+	 (unless (and (let ((blanks (org-element-property
+				     :post-blank special-block)))
+			(or (not blanks) (zerop blanks)))
+		      (eq (org-element-type
+			   (org-export-get-next-element special-block info))
+			  'paragraph))
+	   "</text:p>"))))
      ;; Textbox.
      ((string= type "textbox")
       (let ((width (plist-get attributes :width))
-- 
1.8.2

Re: [BUG] [ODT] Annotations break paragraphs

[Achim Gratz]

Am 25.03.2013 23:15, schrieb Nicolas Goaziou:
>> The proper way to handle this is to introduce a new syntax for
>> inlined annotations and to treat them appropriately in exporters.
>>
>> Since we have both #+begin_src and src_<lang>{...} I'd suggest
>> having annotation_{...} or something similar.
>
> I would suggest [annotation:label] or [note:label] a dedicated section
> for contents, much like footnotes (aren't they just special footnotes,
> after all?). That way, they can be inlined while still being able to
> contain paragraphs.

That only solves this single problem.  Provide something like an 
(implicit) P-Block that will become a paragraph in each exporter and can 
contain other elements like all other blocks.  I suggest to follow the 
lead of (La)TeX and determine begin and end of such blocks by blank lines.

So this would be a single P-Block with an annotation inside:

----8<----
   There is an annotation by the original author here
   #+BEGIN_ANNOTATION
     I never meant to break this paragraph.
   #+END_ANNOTATION
   in the middle of the paragraph.
---->8----

This would be two P-Blocks with an annotation at the end of the first:

----8<----
   There is an annotation by the original author here
   #+BEGIN_ANNOTATION
     I never meant to break this paragraph.
   #+END_ANNOTATION

   But here's the start of a new paragraph.
---->8----


Regards,
-- 
Achim.

(on the road :-)

Re: [BUG] [ODT] Annotations break paragraphs

[Nicolas Goaziou]

Hello,

Achim Gratz <Stromeko@Nexgo.DE> writes:

> Am 25.03.2013 23:15, schrieb Nicolas Goaziou:
>>> The proper way to handle this is to introduce a new syntax for
>>> inlined annotations and to treat them appropriately in exporters.
>>>
>>> Since we have both #+begin_src and src_<lang>{...} I'd suggest
>>> having annotation_{...} or something similar.
>>
>> I would suggest [annotation:label] or [note:label] a dedicated section
>> for contents, much like footnotes (aren't they just special footnotes,
>> after all?). That way, they can be inlined while still being able to
>> contain paragraphs.
>
> That only solves this single problem.  Provide something like an 
> (implicit) P-Block that will become a paragraph in each exporter and can 
> contain other elements like all other blocks.  I suggest to follow the 
> lead of (La)TeX and determine begin and end of such blocks by blank lines.
>
> So this would be a single P-Block with an annotation inside:
>
> ----8<----
>    There is an annotation by the original author here
>    #+BEGIN_ANNOTATION
>      I never meant to break this paragraph.
>    #+END_ANNOTATION
>    in the middle of the paragraph.
> ---->8----

It wouldn't allow paragraphs within the annotation.

Anyway, every back-end has its own interpretation of what a paragraph
is. Some back-ends don't even know what a paragraph is. Org cannot fit
them all.

On the other hand, as the ox-odt patch somehow demonstrates, it is
possible for a back-end to ignore Org paragraph definition and rolls its
own. It requires some additional code, but I'm open to discussion about
implementing tools in ox.el in order to ease the process.

In any case, I think we ought to keep raw Org syntax as simple as
possible. The current definition of a paragraph is simple enough.


Regards,

-- 
Nicolas Goaziou

Re: [BUG] [ODT] Annotations break paragraphs

[Achim Gratz]

Am 27.03.2013 16:48, schrieb Nicolas Goaziou:
>> So this would be a single P-Block with an annotation inside:
>>
>> ----8<----
>>     There is an annotation by the original author here
>>     #+BEGIN_ANNOTATION
>>       I never meant to break this paragraph.
>>     #+END_ANNOTATION
>>     in the middle of the paragraph.
>> ---->8----
>
> It wouldn't allow paragraphs within the annotation.

???

----8<----
     There is an annotation by the original author here
     #+BEGIN_ANNOTATION
       I never meant to break this paragraph.

       But here's a second one in the annotation,
       still not braking the outer paragraph.
     #+END_ANNOTATION
     in the middle of the paragraph.
---->8----

> Anyway, every back-end has its own interpretation of what a paragraph
> is. Some back-ends don't even know what a paragraph is. Org cannot fit
> them all.

That's why Org can't impose its much more restricted paragraph model on 
backends with different paragraph models.

> On the other hand, as the ox-odt patch somehow demonstrates, it is
> possible for a back-end to ignore Org paragraph definition and rolls its
> own. It requires some additional code, but I'm open to discussion about
> implementing tools in ox.el in order to ease the process.

Yes, and footnotes with paragraphs... Anyway, for me this is the main 
sticking point with how Org syntax is defined, because it currently 
implies Org syntax == Backend semantics, which is simply not the case 
for most if not all backends.  Working around this in each and every 
backend doesn't look appealing.

> In any case, I think we ought to keep raw Org syntax as simple as
> possible. The current definition of a paragraph is simple enough.

The syntax wouldn't change all that much, except that blank lines would 
need to be made tokens during parsing.


Regards,
-- 
Achim.

(on the road :-)

Re: [BUG] [ODT] Annotations break paragraphs

[Nicolas Goaziou]

Hello,

Achim Gratz <Stromeko@Nexgo.DE> writes:

>> It wouldn't allow paragraphs within the annotation.
>
> ???
>
> ----8<----
>      There is an annotation by the original author here
>      #+BEGIN_ANNOTATION
>        I never meant to break this paragraph.
>
>        But here's a second one in the annotation,
>        still not braking the outer paragraph.
>      #+END_ANNOTATION
>      in the middle of the paragraph.
> ---->8----

Then you're contradicting yourself, since you also said:

  I suggest to follow the lead of (La)TeX and determine begin and end of
  such blocks by blank lines.

In your example, the end of the "P-block" isn't at the blank line. My
comment was on that precise part of your message.

>> Anyway, every back-end has its own interpretation of what a paragraph
>> is. Some back-ends don't even know what a paragraph is. Org cannot fit
>> them all.
>
> That's why Org can't impose its much more restricted paragraph model on 
> backends with different paragraph models.

As I explained, it doesn't impose anything on back-ends: it merely
offers its reasonable view. You can always ignore its definition, as
written just below.

>> On the other hand, as the ox-odt patch somehow demonstrates, it is
>> possible for a back-end to ignore Org paragraph definition and rolls its
>> own. It requires some additional code, but I'm open to discussion about
>> implementing tools in ox.el in order to ease the process.
>
> Yes, and footnotes with paragraphs... Anyway, for me this is the main 
> sticking point with how Org syntax is defined, because it currently 
> implies Org syntax == Backend semantics, which is simply not the case 
> for most if not all backends.

You're stating the obvious. Org syntax is indeed different from back-end
semantics. Otherwise, we wouldn't need export back-ends to do the
transcoding job, would we?

> Working around this in each and every backend doesn't look appealing.

It didn't prevent dedicated people to rewrite almost all previous
back-ends and implement additional ones.

>> In any case, I think we ought to keep raw Org syntax as simple as
>> possible. The current definition of a paragraph is simple enough.
>
> The syntax wouldn't change all that much, except that blank lines would 
> need to be made tokens during parsing.

This is not only about blank lines. Remember your example:

   There is an annotation by the original author here
   #+BEGIN_ANNOTATION
     I never meant to break this paragraph.
   #+END_ANNOTATION
   in the middle of the paragraph.

There is no blank line, and, yet, it should be parsed differently than
it is actually. Therefore, it's not just about blank lines. It's about
allowing indirectly paragraphs within paragraphs.

Do not misunderstand me: I agree that the current paragraph model is
restricted. But remember we're working with plain text which should be
readable as plain text, not with a full-fledged XML-like markup. The
strength of this paragraph definition lies in its simplicity. And I am
impressed about how much is done with such a simple syntax.

IMO, this kind of syntax, i.e. mixing blocks and paragraphs, is just
ugly. It makes it impossible to read property the paragraph and
emphasizes the annotation, which is counter-productive.

A good rule of thumb is that blocks should be used as containers, never
as inlined elements.

Speaking of inlined elements, I repeat that [note:label] within the
paragraph should offer a good markup for annotations.


Regards,

-- 
Nicolas Goaziou

Re: [BUG] [ODT] Annotations break paragraphs

[Achim Gratz]

Am 28.03.2013 15:41, schrieb Nicolas Goaziou:
> Then you're contradicting yourself, since you also said:
>
>    I suggest to follow the lead of (La)TeX and determine begin and end of
>    such blocks by blank lines.
>
> In your example, the end of the "P-block" isn't at the blank line. My
> comment was on that precise part of your message.

A P-block cannot end inside a special block, just as you don't end a 
paragraph inside an inline footnote.

>> The syntax wouldn't change all that much, except that blank lines would
>> need to be made tokens during parsing.
>
> This is not only about blank lines. Remember your example:
>
>     There is an annotation by the original author here
>     #+BEGIN_ANNOTATION
>       I never meant to break this paragraph.
>     #+END_ANNOTATION
>     in the middle of the paragraph.
>
> There is no blank line, and, yet, it should be parsed differently than
> it is actually. Therefore, it's not just about blank lines. It's about
> allowing indirectly paragraphs within paragraphs.

Well, obviously this example erred on the side of brevity: the begin and 
the end of the document impllicitly begin and end the single P-block, 
which was the point.

----8<----
* Heading
** Sub-Heading
*** Sub-Sub-Heading

     Let's start a single paragraph.

     There is an annotation by the original author here
     #+BEGIN_ANNOTATION
       I never meant to break this paragraph.

       But here's a second one in the annotation,
       still not braking the outer paragraph.
     #+END_ANNOTATION
     in the middle of the paragraph.

     Here's another paragraph.
---->8----

This would be parsed as:

----8<----
<heading level 1>
<heading level 2>
<heading level 3>
begin_p-block
   <paragraph>
end_p-block
begin_p-block
   <paragraph>
   <annotation>
   <paragraph>
end_p-block
begin_p-block
   <paragraph>
end_p-block
---->8----

> Do not misunderstand me: I agree that the current paragraph model is
> restricted. But remember we're working with plain text which should be
> readable as plain text, not with a full-fledged XML-like markup. The
> strength of this paragraph definition lies in its simplicity. And I am
> impressed about how much is done with such a simple syntax.

Again and maybe I'm expressing myself too poorly, I don't suggest to 
change that model.  It is working well for Org as far as I can tell. 
What I'm suggesting is an extension of that model, ostensibly to help 
supporting more complex models for exporting.  To this end blank lines 
must either be handed explicitly to the exporter and not gobbled up by 
org-element or org-element could extract P-blocks as shown above.

> IMO, this kind of syntax, i.e. mixing blocks and paragraphs, is just
> ugly. It makes it impossible to read property the paragraph and
> emphasizes the annotation, which is counter-productive.

What about tables, images, listings, equations?  You already have that 
very problem (each of those break the paragraph around them) in many 
forms.  You can fix them one by one and tweak heuristics in the exporter 
for the next two years or you can provide a general mechanism to deal 
with that situation.

> A good rule of thumb is that blocks should be used as containers, never
> as inlined elements.

That's why the exported paragraph must be a separate entity from what 
Org calls a paragraph: there are many block elements in Org that can and 
should possibly be expüorted as inlined elements in the what gets 
exported as a paragraph.

> Speaking of inlined elements, I repeat that [note:label] within the
> paragraph should offer a good markup for annotations.

While I agree that this is a good solution for annotations and mimics 
the way inline footnotes work, it introduces a new syntactical element 
only for escaping that limitation.  Now, when you think about LaTeX 
blocks you can't treat them the same easily, but there really is no good 
reason why an equation array should introduce an unwanted paragraph 
boundary in LaTeX (again, I'm not talking about the paragraph boundaries 
in Org).


Regards,
-- 
Achim.

(on the road :-)

-- 
Achim.

(on the road :-)

Re: [BUG] [ODT] Annotations break paragraphs

[Christian Moe]

> Speaking of inlined elements, I repeat that [note:label] within the
> paragraph should offer a good markup for annotations.

I agree, and favor a move from annotation blocks to a solution like
that.[note:1]

[note:1] That is, like this?

One thing to consider is whether this markup should support author and
timestamp info in a way that the parser should understand and pass on to
relevant backends. The annotation blocks do[note:1], because ODT does.

[note:2] C.Moe <2013-03-28 Thu 21:34>: Except that they don't properly
  support it at the moment, as I brought up in another message.

One could even use the author/timestamp information to form unique
labels,[note:C.Moe_2013-03-28_21:37:26] but the labels would get long
and messy.

[note:C.Moe_2013-03-28_21:37:26] Like this. 

Yours,
Christian

Re: [BUG] [ODT] Annotations break paragraphs

[Christian Moe]

Ouch, I messed those illustrations up, and I suppose my message wasn't
easy to understand to begin with. Sorry.

> One thing to consider is whether this markup should support author and
> timestamp info in a way that the parser should understand and pass on to
> relevant backends. The annotation blocks do[note:1], because ODT does.

For "[note:1]" in the above line, read "[note:2]"...

Christian

Re: [BUG] [ODT] Annotations break paragraphs

[Christian Moe]

Hi,

Patch applied and tested -- looks good! Thanks. 

Annotations now work again both inside and at the end of paragraphs and
list items, without introducing unwanted paragraph breaks or swallowing
wanted ones. I haven't looked specifically for corner cases, but I've
successfully exported a fairly long document with lots of annotations.

Minor cosmetic annoyance: An extra space is inserted
before the annotation.

Pending the introduction of a dedicated syntax, could we have this fix
in master please?

Yours,
Christian




Nicolas Goaziou writes:

> Hello,
>
> Bastien <bzg@altern.org> writes:
>
>> okay, I reverted my wrong fixes.  I'll let Nicolas have a look.
>
> I attach a patch (hardly tested) for that. Does it work as expected?
>
>> I would not favor a solution that allows more #+begin_ blocks to
>> be inlined.
>
> Neither would I. Blocks are containers.
>
>> The proper way to handle this is to introduce a new syntax for
>> inlined annotations and to treat them appropriately in exporters.
>>
>> Since we have both #+begin_src and src_<lang>{...} I'd suggest
>> having annotation_{...} or something similar.
>
> I would suggest [annotation:label] or [note:label] a dedicated section
> for contents, much like footnotes (aren't they just special footnotes,
> after all?). That way, they can be inlined while still being able to
> contain paragraphs.
>
>> The LaTeX exporter could use \marginpar{...} and the HTML back-end
>> could make them appear when hovering with the mouse on the annotated
>> parts (just an idea.)
>>
>> Maybe we will have to live with the current "regression" for 8.0
>> and implement the new syntax for 8.1.  Or for 8.0, if Nicolas thinks
>> the change is okay and not too error prone.
>
>
> Regards,

Re: [BUG] [ODT] Annotations break paragraphs

[Nicolas Goaziou]

Hello,

Christian Moe <mail@christianmoe.com> writes:

> Patch applied and tested -- looks good! Thanks. 
>
> Annotations now work again both inside and at the end of paragraphs and
> list items, without introducing unwanted paragraph breaks or swallowing
> wanted ones. I haven't looked specifically for corner cases, but I've
> successfully exported a fairly long document with lots of annotations.
>
> Minor cosmetic annoyance: An extra space is inserted
> before the annotation.
>
> Pending the introduction of a dedicated syntax, could we have this fix
> in master please?

Sure, but, if it's not too much work, I'd like the extra space removed
first. Do you have an idea about the XML reason behind this?


Regards,

-- 
Nicolas Goaziou

Re: [BUG] [ODT] Annotations break paragraphs

[Christian Moe]

Nicolas Goaziou writes:
>> Minor cosmetic annoyance: An extra space is inserted
>> before the annotation.
>>
>> Pending the introduction of a dedicated syntax, could we have this fix
>> in master please?
>
> Sure, but, if it's not too much work, I'd like the extra space removed
> first. Do you have an idea about the XML reason behind this?

Yes, there's a newline inserted before the opening <office:annotation>
tag. I guess that since it's inside a <text:p> tag, it gets interpreted
as whitespace that's part of the text, and collapsed to a space. But the
newline is only there to make the XML more readable, so it's expendable.

Sorry, I really could have taken the ten minutes to investigate that
before my previous message.

Yours,
Christian

Re: [BUG] [ODT] Annotations break paragraphs

[Christian Moe]

Hi again,

While we're at it, there's trouble with custom author and date info in
annotations. I hadn't really noticed, since my name was inserted anyway
and dates didn't matter much to my work.

1. Author and date attributes of the annotation are ignored, unless I'm
writing them wrong (I've tried three syntaxes -- below the
signature). My name is used for the authorname, whatever I put as
:author in the #+begin_annotation line. No date is found.

2. In LibreOffice the annotations all say "no date", and indeed, no date
(<dc:date> element) is inserted at all. Maybe it should be. When I
remove the condition testing for a date "(and date <...>)", all the
annotations get stamped with the time of export if the exporter doesn't
find any other date. That was the behavior of the old exporter. It's
debatable whether it's the right thing to do, but I saw it as a feature,
saving work to insert timestamps.

Yours,
Christian

Here's what I've tried for the author/date attributes.

This is the syntax originally used when Jambunathan introduced the feature:

There is an annotation by a reviewer - "Someone else" - here.
#+begin_annotation author "Someone else" date "<2011-10-22 Wed 15:43>"
Yes I do.
#+end_annotation

I have tried these as well:

There is an annotation by a reviewer - "Someone else" - here.
#+begin_annotation :author "Someone else" :date "<2011-10-22 Wed 15:43>"
Yes I do.
#+end_annotation

There is an annotation by a reviewer - "Someone else" - here.
#+begin_annotation :author Someone else :date <2011-10-22 Wed 15:43>
Yes I do.
#+end_annotation



Christian Moe writes:

> Nicolas Goaziou writes:
>>> Minor cosmetic annoyance: An extra space is inserted
>>> before the annotation.
>>>
>>> Pending the introduction of a dedicated syntax, could we have this fix
>>> in master please?
>>
>> Sure, but, if it's not too much work, I'd like the extra space removed
>> first. Do you have an idea about the XML reason behind this?
>
> Yes, there's a newline inserted before the opening <office:annotation>
> tag. I guess that since it's inside a <text:p> tag, it gets interpreted
> as whitespace that's part of the text, and collapsed to a space. But the
> newline is only there to make the XML more readable, so it's expendable.
>
> Sorry, I really could have taken the ten minutes to investigate that
> before my previous message.
>
> Yours,
> Christian

Re: [BUG] [ODT] Annotations break paragraphs

[Christian Moe]

Hi Bastien,

Thanks for giving it a try. Let's see what Nicolas comes up with.

>
> I would not favor a solution that allows more #+begin_ blocks to
> be inlined.

We've had this one for a while. It's always been a bit anomalous though.

> The proper way to handle this is to introduce a new syntax for
> inlined annotations and to treat them appropriately in exporters.

I think I'd agree.

> Since we have both #+begin_src and src_<lang>{...} I'd suggest
> having annotation_{...} or something similar.

But if annotations are not meant to live in blocks anyway, that analogy
fails.

Aren't annotations/comments more analogous to footnotes? They expand on
the text, but you don't want to clutter the text with them. Could we
imagine e.g. an [cmt:: This is an inline comment] syntax? (It would be
good to support timestamps and authornames, though. Maybe colons aren't
good separators if we put timestamps inside. Hm, needs thinking about.)

> The LaTeX exporter could use \marginpar{...} and the HTML back-end
> could make them appear when hovering with the mouse on the annotated
> parts (just an idea.)

Whatever the solution it would be good to have it work for all backends
where it makes sense.

> Maybe we will have to live with the current "regression" for 8.0
> and implement the new syntax for 8.1.  Or for 8.0, if Nicolas thinks
> the change is okay and not too error prone.

I can always go back to inserting ODT annotations with macros, for the
time being.

Yours,
Christian