Problem with org-export--delete-comments

Problem with org-export--delete-comments

[Ethan Ligon]

I've long used the following construction for displayed equations in org
#
\[
    u_i(c)=p_i\lambda
\]
#
which (i) gives nice space for reading equations in the org-source, and 
(ii) nicely protects the display equation from (fill-paragraph) and 
friends.

However, exporting with this construction *stopped* working after commit 
53a4209; what happens now is that ox.el replaces the # with a blank line, 
breaking my single paragraph into  three.

A related problem: one can no longer have commented lines in (at least 
some) property drawers.

It seems to me that the correct solution would have org-export--delete-
comments actually delete a single line comment, not replace it with  \n, as 
was the case prior to  the  indicated commit.

I gather that the change was introduced to deal with a bug observed by 
Samuel Wales (http://article.gmane.org/gmane.emacs.orgmode/102860; see also  
below).  His problem example involved incorrectly truncated footnotes 
(these are deemed to end after two blank lines).  But it seems to me that 
the correct solution to his example is to  delete *both* the  commented 
line and the  following blank line.

I regard this as a bug, but if it's not how do others deal with similar 
issues?

Thanks,
-Ethan Ligon

* Sam Wales' example: commenting a paragraph will truncate footnotes
in xanadu[fn:b3e0f07]

um

# comment

[fn:b3e0f07]
did kublai khan

# was it "seemly"?

a stately pleasure dome

decree

Re: Problem with org-export--delete-comments

[Nicolas Goaziou]

Hello,

Ethan Ligon <ligon@berkeley.edu> writes:

> I've long used the following construction for displayed equations in org
> #
> \[
>     u_i(c)=p_i\lambda
> \]
> #
> which (i) gives nice space for reading equations in the org-source, and 
> (ii) nicely protects the display equation from (fill-paragraph) and 
> friends.
>
> However, exporting with this construction *stopped* working after commit 
> 53a4209; what happens now is that ox.el replaces the # with a blank line, 
> breaking my single paragraph into  three.

I see only one paragraph above. Could you elaborate a bit?

> A related problem: one can no longer have commented lines in (at least 
> some) property drawers.

This is not possible indeed, per property drawers syntax. Also, I'd
rather not introduce comments in property drawers (I don't think this
was ever officially supported anyway).

> It seems to me that the correct solution would have org-export--delete-
> comments actually delete a single line comment, not replace it with  \n, as 
> was the case prior to  the  indicated commit.
>
> I gather that the change was introduced to deal with a bug observed by 
> Samuel Wales (http://article.gmane.org/gmane.emacs.orgmode/102860; see also  
> below).  His problem example involved incorrectly truncated footnotes 
> (these are deemed to end after two blank lines).  But it seems to me that 
> the correct solution to his example is to  delete *both* the  commented 
> line and the  following blank line.

Been there, done that. Your solution is not correct in all cases. E.g.,
with the following document

  Paragra1
  # comment

  Paragraph2

removing both the comment and the blank line above gives you a single
paragraph, which doesn't match expectations.

> I regard this as a bug, but if it's not how do others deal with similar 
> issues?

I have the feeling that there is no perfect solution here, and something
will break on one side or the other, anyway.


Regards,

-- 
Nicolas Goaziou

Re: Problem with org-export--delete-comments

[Andreas Leha]

Hi Nicolas and Ethan,

Nicolas Goaziou <mail@nicolasgoaziou.fr> writes:
> Hello,
>
> Ethan Ligon <ligon@berkeley.edu> writes:
>
>> I've long used the following construction for displayed equations in org
>> #
>> \[
>>     u_i(c)=p_i\lambda
>> \]
>> #
>> which (i) gives nice space for reading equations in the org-source, and 
>> (ii) nicely protects the display equation from (fill-paragraph) and 
>> friends.
>>
>> However, exporting with this construction *stopped* working after commit 
>> 53a4209; what happens now is that ox.el replaces the # with a blank line, 
>> breaking my single paragraph into  three.
>
> I see only one paragraph above. Could you elaborate a bit?

I think the problem is in the exported latex file rather than in the org
document.  While the org document indeed has one paragraph the exported
latex file does not.  I can reproduce this.

With this org document:

--8<---------------cut here---------------start------------->8---
* Test

I've long used the following construction for displayed equations in org
#
\[
    u_i(c)=p_i\lambda
\]
#
which (i) gives nice space for reading equations in the org-source, and
(ii) nicely protects the display equation from (fill-paragraph) and
friends.
--8<---------------cut here---------------end--------------->8---


I get this latex doc (relevant part only):

--8<---------------cut here---------------start------------->8---
\section{Test}
\label{sec:orgheadline1}

I've long used the following construction for displayed equations in org

\[
    u_i(c)=p_i\lambda
\]

which (i) gives nice space for reading equations in the org-source, and
(ii) nicely protects the display equation from (fill-paragraph) and
friends.
--8<---------------cut here---------------end--------------->8---

I agree that the paragraph should not be broken in the exported
document.  There are also other use cases for comments inside
paragraphs.

[ ... ]

Regards,
Andreas

Re: Problem with org-export--delete-comments

[Nicolas Goaziou]

Hello,

Andreas Leha <andreas.leha@med.uni-goettingen.de> writes:

> I agree that the paragraph should not be broken in the exported
> document.  There are also other use cases for comments inside
> paragraphs.

Note that Org doesn't support comments within paragraphs since a comment
ends a paragraph. In this case, the export output matches the contents
of the document.

Again, I'm not sure there is a solution satisfying all needs here. 

However, you can mimic inline comments with export snippets, e.g.,
@@comment:...@@.

Regards,

-- 
Nicolas Goaziou

Re: Problem with org-export--delete-comments

[Andreas Leha]

Hi Nicolas,

Nicolas Goaziou <mail@nicolasgoaziou.fr> writes:
> Hello,
>
> Andreas Leha <andreas.leha@med.uni-goettingen.de> writes:
>
>> I agree that the paragraph should not be broken in the exported
>> document.  There are also other use cases for comments inside
>> paragraphs.
>
> Note that Org doesn't support comments within paragraphs since a comment
> ends a paragraph. In this case, the export output matches the contents
> of the document.
>
> Again, I'm not sure there is a solution satisfying all needs here. 
>

I do not really follow the footnote problem (not using footnotes much)
with this.  But if removing the line following the comment works, then
I'd suggest to simply also comment that newline.

In my opinion comments are not newlines.  Exporting comment lines as
newlines makes it impossible to comment newlines, which might be
desirable -- see this thread.

But I might be missing the point here.

> However, you can mimic inline comments with export snippets, e.g.,
> @@comment:...@@.

That is handy.  I would not have thought of that.

Regards,
Andreas

Re: Problem with org-export--delete-comments

[Ethan Ligon]

Nicolas Goaziou <mail <at> nicolasgoaziou.fr> writes:
> 
> Note that Org doesn't support comments within paragraphs since a comment
> ends a paragraph. In this case, the export output matches the contents
> of the document.
> 

It seems odd that a comment ends a paragraph, but does not end a footnote!

-Ethan

Re: Problem with org-export--delete-comments

[Nicolas Goaziou]

Hello,

Ethan Ligon <ligon@berkeley.edu> writes:

> It seems odd that a comment ends a paragraph, but does not end
> a footnote!

Not really, paragraphs and footnote definitions are different beast,
since the latter can contain the former.

Regards,

-- 
Nicolas Goaziou

Re: Problem with org-export--delete-comments

[Ethan Ligon]

[-- Attachment #1: Type: text/plain, Size: 3551 bytes --]

On Thu, Dec 17, 2015 at 12:47 PM Nicolas Goaziou <mail@nicolasgoaziou.fr>
wrote:

> Hello,
>
> Ethan Ligon <ligon@berkeley.edu> writes:
>
> > I've long used the following construction for displayed equations in org
> > #
> > \[
> >     u_i(c)=p_i\lambda
> > \]
> > #
> > which (i) gives nice space for reading equations in the org-source, and
> > (ii) nicely protects the display equation from (fill-paragraph) and
> > friends.
> >
> > However, exporting with this construction *stopped* working after commit
> > 53a4209; what happens now is that ox.el replaces the # with a blank line,
> > breaking my single paragraph into  three.
>
> I see only one paragraph above. Could you elaborate a bit?
>

Sure.  If I export the above using ox-ascii, I obtain, in part:

#+begin_example
I've long used the following construction for displayed equations in org

\[ u_i(c)=p_i\lambda \]

which (i) gives nice space for reading equations in the org-source, and
(ii) nicely protects the display equation from (fill-paragraph) and
friends.
#+end_example

While if  I export via ox-latex I obtain in the *.tex source:

#+begin_example
I've long used the following construction for displayed equations in org

\[
    u_i(c)=p_i\lambda
\]

which (i) gives nice space for reading equations in the org-source, and
(ii) nicely protects the display equation from (fill-paragraph) and
friends.
#+end_example

It's really this last case that's a serious problem, since when compiled
with  LaTeX we get three paragraphs (too much vertical space between  the
 first line and the displayed equation, and excruciatingly an improper
indentation of the bit following the  displayed equation (i.e., "which
(i)..." is indented).





>
> > A related problem: one can no longer have commented lines in (at least
> > some) property drawers.
>
> This is not possible indeed, per property drawers syntax. Also, I'd
> rather not introduce comments in property drawers (I don't think this
> was ever officially supported anyway).
>
> > It seems to me that the correct solution would have org-export--delete-
> > comments actually delete a single line comment, not replace it with  \n,
> as
> > was the case prior to  the  indicated commit.
> >
> > I gather that the change was introduced to deal with a bug observed by
> > Samuel Wales (http://article.gmane.org/gmane.emacs.orgmode/102860; see
> also
> > below).  His problem example involved incorrectly truncated footnotes
> > (these are deemed to end after two blank lines).  But it seems to me that
> > the correct solution to his example is to  delete *both* the  commented
> > line and the  following blank line.
>
> Been there, done that. Your solution is not correct in all cases. E.g.,
> with the following document
>
>   Paragra1
>   # comment
>
>   Paragraph2
>
> removing both the comment and the blank line above gives you a single
> paragraph, which doesn't match expectations.
>
> Point taken,  and I appreciate that you must have thought about this quite
a lot.   But I still wish for the old behavior, which  would have simply
deleted the # comment line.

This would break SW's footnote example, but perhaps that (contrived? maybe
I don't understand his  use case) problem could be addressed by changing
the documentation.  Right now the docs say that a footnote

"ends at the next footnote definition, headline, or after two consecutive
empty lines."

 Why not just change to

"ends at the next footnote definition, headline, or two empty lines
(consecutive, or with a comment)."

Thanks for your work on this!
-Ethan

[-- Attachment #2: Type: text/html, Size: 5395 bytes --]

Re: Problem with org-export--delete-comments

[Nicolas Goaziou]

Hello,

Ethan Ligon <ligon@are.berkeley.edu> writes:

> On Thu, Dec 17, 2015 at 12:47 PM Nicolas Goaziou <mail@nicolasgoaziou.fr>
> wrote:
>
>> Hello,
>>
>> Ethan Ligon <ligon@berkeley.edu> writes:
>>
>> > I've long used the following construction for displayed equations in org
>> > #
>> > \[
>> >     u_i(c)=p_i\lambda
>> > \]
>> > #
>> > which (i) gives nice space for reading equations in the org-source, and
>> > (ii) nicely protects the display equation from (fill-paragraph) and
>> > friends.
>> >
>> > However, exporting with this construction *stopped* working after commit
>> > 53a4209; what happens now is that ox.el replaces the # with a blank line,
>> > breaking my single paragraph into  three.
>>
>> I see only one paragraph above. Could you elaborate a bit?
>>
>
> Sure.  If I export the above using ox-ascii, I obtain, in part:
>
> #+begin_example
> I've long used the following construction for displayed equations in org
>
> \[ u_i(c)=p_i\lambda \]
>
> which (i) gives nice space for reading equations in the org-source, and
> (ii) nicely protects the display equation from (fill-paragraph) and
> friends.
> #+end_example

Actually this is consistent with the Org document. As explained
somewhere else in this thread, comments separate elements (e.g.,
paragraphs). They cannot be inlined within a paragraph. This is exactly
what happens in the output.

I think you are simply mis-using comments in this particular case.

> Point taken,  and I appreciate that you must have thought about this quite
s> a lot.   But I still wish for the old behavior, which  would have simply
> deleted the # comment line.
>
> This would break SW's footnote example, but perhaps that (contrived? maybe
> I don't understand his  use case) problem could be addressed by changing
> the documentation.  Right now the docs say that a footnote
>
> "ends at the next footnote definition, headline, or after two consecutive
> empty lines."
>
>  Why not just change to
>
> "ends at the next footnote definition, headline, or two empty lines
> (consecutive, or with a comment)."

This is not related to footnote syntax. We could find a similar example
in, e.g., plain lists. The problem is simply that comment syntax doesn't
fit for your use-case.

The simplest solution is to write a function handling commented lines
the way you want them to be handled and add it to
`org-export-before-processing-hook', e.g. (untested),

  (defun my-handle-comments ()
    (while (re-search-forward "^[ \t]*#\\( \\|$\\)" nil t)
      (let ((element (org-element-at-point)))
        (when (eq (org-element-type element) 'comment)
          (delete-region (org-element-property :begin element)
                         (progn (goto-char (org-element-property :end element))
                                (skip-chars-backward " \t\n")
                                (line-beginning-position 2)))))))


Also, I have a dusty patch somewhere that promotes \[...\] and $$...$$
to the rank of elements and prevent them from being filled. However, it
also means that these constructs are not recognized as math snippets
within a line anymore, e.g.,

  Text \[1+1\] ...

I'm not sure how much of an issue it is. Hence the dust.


Regards,

-- 
Nicolas Goaziou

Re: Problem with org-export--delete-comments

[Ethan Ligon]

[-- Attachment #1: Type: text/plain, Size: 3957 bytes --]

On Sun, Dec 20, 2015 at 1:29 AM Nicolas Goaziou <mail@nicolasgoaziou.fr>
wrote:

> Hello,
>
> Ethan Ligon <ligon@are.berkeley.edu> writes:
>
> > On Thu, Dec 17, 2015 at 12:47 PM Nicolas Goaziou <mail@nicolasgoaziou.fr
> >
> > wrote:
> >
> >> Hello,
> >>
> >> Ethan Ligon <ligon@berkeley.edu> writes:
> >>
> >> > I've long used the following construction for displayed equations in
> org
> >> > #
> >> > \[
> >> >     u_i(c)=p_i\lambda
> >> > \]
> >> > #
> >> > which (i) gives nice space for reading equations in the org-source,
> and
> >> > (ii) nicely protects the display equation from (fill-paragraph) and
> >> > friends.
> >> >
> >> > However, exporting with this construction *stopped* working after
> commit
> >> > 53a4209; what happens now is that ox.el replaces the # with a blank
> line,
> >> > breaking my single paragraph into  three.
> >>
> >> I see only one paragraph above. Could you elaborate a bit?
> >>
> >
> > Sure.  If I export the above using ox-ascii, I obtain, in part:
> >
> > #+begin_example
> > I've long used the following construction for displayed equations in org
> >
> > \[ u_i(c)=p_i\lambda \]
> >
> > which (i) gives nice space for reading equations in the org-source, and
> > (ii) nicely protects the display equation from (fill-paragraph) and
> > friends.
> > #+end_example
>
> Actually this is consistent with the Org document. As explained
> somewhere else in this thread, comments separate elements (e.g.,
> paragraphs). They cannot be inlined within a paragraph. This is exactly
> what happens in the output.
>
> I think you are simply mis-using comments in this particular case.
>
> > Point taken,  and I appreciate that you must have thought about this
> quite
> s> a lot.   But I still wish for the old behavior, which  would have simply
> > deleted the # comment line.
> >
> > This would break SW's footnote example, but perhaps that (contrived?
> maybe
> > I don't understand his  use case) problem could be addressed by changing
> > the documentation.  Right now the docs say that a footnote
> >
> > "ends at the next footnote definition, headline, or after two consecutive
> > empty lines."
> >
> >  Why not just change to
> >
> > "ends at the next footnote definition, headline, or two empty lines
> > (consecutive, or with a comment)."
>
> This is not related to footnote syntax. We could find a similar example
> in, e.g., plain lists. The problem is simply that comment syntax doesn't
> fit for your use-case.
>

That's fair.  That was simply  a work-around I'd developed that worked
until a few weeks ago, to solve the deeper problem you  identify below...

>
> The simplest solution is to write a function handling commented lines
> the way you want them to be handled and add it to
> `org-export-before-processing-hook', e.g. (untested),
>
>   (defun my-handle-comments ()
>     (while (re-search-forward "^[ \t]*#\\( \\|$\\)" nil t)
>       (let ((element (org-element-at-point)))
>         (when (eq (org-element-type element) 'comment)
>           (delete-region (org-element-property :begin element)
>                          (progn (goto-char (org-element-property :end
> element))
>                                 (skip-chars-backward " \t\n")
>                                 (line-beginning-position 2)))))))
>

Thanks!  I really appreciate the help.

>
> Also, I have a dusty patch somewhere that promotes \[...\] and $$...$$
> to the rank of elements and prevent them from being filled. However, it
> also means that these constructs are not recognized as math snippets
> within a line anymore, e.g.,
>
>   Text \[1+1\] ...
>
> I'm not sure how much of an issue it is. Hence the dust.
>
> Well, I most certainly vote for applying your dusty patch.  In my
(admittedly LaTeX-centric) view the whole point of the  distinction between
\(\) and \[\] is that the former is meant to be an in-line construct, while
the latter is meant to indicate a typographically distinct element.

Thanks,
-Ethan

[-- Attachment #2: Type: text/html, Size: 5402 bytes --]