Inserting a comma as prefix of headlines (in Org code blocks)

Inserting a comma as prefix of headlines (in Org code blocks)

[Sebastien Vauban]

Hello,

Consider this Org block:

#+begin_src org
* Headline inside an Org block

Text of that entry.
#+end_src

I don't remember exactly why (I think, sometimes, TAB'ing did not work
properly to prefix the dangerous lines with a `,'), I had to do it manually:
go and prefix every headline of the Org code block.

#+begin_src org
,* Headline inside an Org block

Text of that entry.
#+end_src

That worked for long, but not anymore since a couple of weeks (could be
something like 2 months or so): currently, if you go in column 0 of the
headline, and insert `,', you get asked:

  "Priority A-C, SPC to remove: "

IMHO, such a question should not be asked, and the `,' should be inserted
literally, as we're in the context of a code block.

Best regards,
  Seb

-- 
Sebastien Vauban

Re: Inserting a comma as prefix of headlines (in Org code blocks)

[Bastien]

Hi Sébastien,

"Sebastien Vauban"
<wxhgmqzgwmuf-geNee64TY+gS+FvcfC7Uqw@public.gmane.org> writes:

> IMHO, such a question should not be asked, and the `,' should be inserted
> literally, as we're in the context of a code block.

Fixed in master, thanks.

-- 
 Bastien

Re: Inserting a comma as prefix of headlines (in Org code blocks)

[Nicolas Goaziou]

Hello,

"Sebastien Vauban"
<wxhgmqzgwmuf-geNee64TY+gS+FvcfC7Uqw@public.gmane.org> writes:

> Consider this Org block:
>
> #+begin_src org
> * Headline inside an Org block
>
> Text of that entry.
> #+end_src

Technically, this is not an Org block for any serious function in Org.
Headlines have precedence over block markers.

> I don't remember exactly why (I think, sometimes, TAB'ing did not work
> properly to prefix the dangerous lines with a `,'), I had to do it manually:
> go and prefix every headline of the Org code block.

> #+begin_src org
> ,* Headline inside an Org block
>
> Text of that entry.
> #+end_src
>
> That worked for long, but not anymore since a couple of weeks (could be
> something like 2 months or so): currently, if you go in column 0 of the
> headline, and insert `,', you get asked:
>
>   "Priority A-C, SPC to remove: "
>
> IMHO, such a question should not be asked, and the `,' should be inserted
> literally, as we're in the context of a code block.

We're not, that's why comma escaping exists in the first place.

I understand this may be counter-intuitive in your simple example, but
it's quite logical.

I think a proper solution would be to remove "," from speed keys and put
priority command elsewhere. Thus, you can always insert "," manually at
the beginning of the line. Meanwhile, there is "C-q ,".n

What about "#" for priorities?


Regards,

-- 
Nicolas Goaziou

Re: Inserting a comma as prefix of headlines (in Org code blocks)

[Bastien]

Hi Nicolas,

Nicolas Goaziou <n.goaziou@gmail.com> writes:

> What about "#" for priorities?

"#" is already taken, for comments.

I think "," is good for priorities, and that preventing speed commands
in the several blocks is safe and non-intrusive, that's what my patch
did.  Let me know if you (strongly) think otherwise!

-- 
 Bastien

Re: Inserting a comma as prefix of headlines (in Org code blocks)

[Nicolas Goaziou]

Hello,

Bastien <bzg@gnu.org> writes:

> "#" is already taken, for comments.

Ok then another binding. I still think freeing "," key is the best thing
to do. More on this below.

> I think "," is good for priorities, and that preventing speed commands
> in the several blocks is safe and non-intrusive, that's what my patch
> did. Let me know if you (strongly) think otherwise!

Well, yes, I strongly think otherwise.

Your patch is relying on `org-in-block-p', which is completely broken in
this situation.

The fact is that any strictly positive number of "*" at column
0 followed by a space define a headline, whatever the context is. In
other words, headlines have precedence over every other construct in Org
syntax. 

It's not about the parser. Every low level Org command (and most of the
high level too) assume, and have always assumed, this. For example, try
to cycle visibility in the following example (or move forward
heading...):

--8<---------------cut here---------------start------------->8---
* H1

** H11

#+begin_example
** H12
#+end_example
--8<---------------cut here---------------end--------------->8---

So, we have to make this point clear once and for all. Otherwise, we
should as well re-implement all functions working on headlines, because
if we accept that (org-in-block-p '("example")) returns a non-nil value
in the previous example, they become all wrong.

Therefore, "** H12" is a headline, and it is to be expected that speed
commands are triggered when point is on column 0. So my suggested
solution for the problem at hand is to simply avoid using "," in speed
keys, since "," is also used to protect headlines in blocks.

Let me stress this again:

  1. "stars + space" at column 0 define a headline. No exception. Most
     of Org code (reasonably) assumes this, so we should not let users
     think otherwise.

  2. Do not rely on `org-in-block-p'. Please use `org-element-at-point'
     or `org-element-context' instead. These are not broken, and they
     are fast enough for any interactive use (but let's not use them for
     fontification yet).


Regards,

-- 
Nicolas Goaziou

Re: Inserting a comma as prefix of headlines (in Org code blocks)

[Bastien]

Hi Nicolas,

Nicolas Goaziou <n.goaziou@gmail.com> writes:

> Ok then another binding. I still think freeing "," key is the best thing
> to do. More on this below.

Users will still be able to use the "," so this will not really fix
the issue.

>> I think "," is good for priorities, and that preventing speed commands
>> in the several blocks is safe and non-intrusive, that's what my patch
>> did. Let me know if you (strongly) think otherwise!
>
> Well, yes, I strongly think otherwise.
>
> Your patch is relying on `org-in-block-p', which is completely broken in
> this situation.
>
> The fact is that any strictly positive number of "*" at column
> 0 followed by a space define a headline, whatever the context is. In
> other words, headlines have precedence over every other construct in Org
> syntax. 
>
> It's not about the parser. Every low level Org command (and most of the
> high level too) assume, and have always assumed, this. For example, try
> to cycle visibility in the following example (or move forward
> heading...):
>
> * H1
>
> ** H11
>
> #+begin_example
> ** H12
> #+end_example
>
> So, we have to make this point clear once and for all. Otherwise, we
> should as well re-implement all functions working on headlines, because
> if we accept that (org-in-block-p '("example")) returns a non-nil value
> in the previous example, they become all wrong.

I got your point, but "broken" is contextual.

The fact that (org-in-block-p '("src")) returns a non-nil value in

#+begin_src org
** H12
#+end_src

is not wrong in the context of checking whether a speed command should
be prevented or not.  It might be wrong in other contexts, but for
this purpose it is not.

That's similar to TAB, which comma-escapes the content of the block
instead of cycling through the folding states, because it knows it is
in a src block.

>   1. "stars + space" at column 0 define a headline. No exception. Most
>      of Org code (reasonably) assumes this, so we should not let users
>      think otherwise.

Yes.  But it is not because the cursor is at the beginning of a
headline that every function should behave the same.  TAB does not,
speed commands do not either.

>   2. Do not rely on `org-in-block-p'. Please use `org-element-at-point'
>      or `org-element-context' instead. These are not broken, and they
>      are fast enough for any interactive use (but let's not use them for
>      fontification yet).

Btw, can you think of cases where it would be nice to have
`org-element-context' check against a wider context than the closest
one?

-- 
 Bastien

Re: Inserting a comma as prefix of headlines (in Org code blocks)

[Nicolas Goaziou]

Bastien <bzg@altern.org> writes:

> Nicolas Goaziou <n.goaziou@gmail.com> writes:
>
>> Ok then another binding. I still think freeing "," key is the best thing
>> to do. More on this below.
>
> Users will still be able to use the "," so this will not really fix
> the issue.

That's the whole point: leave the ability to escape the headline
manually (without using "C-q ,"), as Org should be unable to do so.

> The fact that (org-in-block-p '("src")) returns a non-nil value in
>
> #+begin_src org
> ** H12
> #+end_src
>
> is not wrong in the context of checking whether a speed command should
> be prevented or not.  It might be wrong in other contexts, but for
> this purpose it is not.

It is wrong because there's no block here, although it finds one.
Functions should not be built on erroneous assumptions like this.

> That's similar to TAB, which comma-escapes the content of the block
> instead of cycling through the folding states, because it knows it is
> in a src block.

It cannot know. If there's a headline amidst a "block", that "block"
dissolves because its opening line is separated from the closing one.

Again, headlines are first-class objects in Org. A command shouldn't
pretend there is no headline when point is on one. TAB has to cycle in
this case (like it does in an example block, for example), even if it
looks strange on trivial examples.

>>   1. "stars + space" at column 0 define a headline. No exception. Most
>>      of Org code (reasonably) assumes this, so we should not let users
>>      think otherwise.
>
> Yes.  But it is not because the cursor is at the beginning of a
> headline that every function should behave the same.  TAB does not,
> speed commands do not either.

Since headlines are the base of Org syntax, I think it should be
expected that commands always behave consistently on them.

There is no more important context than a headline. Ignoring it may look
useful on a couple of examples, but we're shooting ourselves in the
foot. Think about it, for example:

  #+begin_src something
  ...
  more than one screenful of lines
  ...
  * Headline
  ...
  more than one screenful of lines
  #+end_src

Appending a comma to "* Headline" after pressing TAB may be surprising
if the user has forgotten about the context.

Another example:

  #+BEGIN_src emacs-lisp
  * H1
  ** H11
  #+END_src

Press TAB on any of the headlines. It will neither cycle them nor add
a comma in front of them.

This hack is not worth it. It's far better to have a simple rule to
handle them: "If you need to have a star at column 0 in a block, put
a comma before it.". One line in the manual is better than scores of
loc, which can always be fooled under certain circumstances.

>>   2. Do not rely on `org-in-block-p'. Please use `org-element-at-point'
>>      or `org-element-context' instead. These are not broken, and they
>>      are fast enough for any interactive use (but let's not use them for
>>      fontification yet).
>
> Btw, can you think of cases where it would be nice to have
> `org-element-context' check against a wider context than the closest
> one?

`org-element-context' already provides access to wider context
through :parent property. In the following example:

  *bold _and_ underlined text*

with, point on "and",

  (org-element-type (org-element-property :parent (org-element-context)))

will return `bold' and

  (org-element-type
    (org-element-property :parent
      (org-element-property :parent (org-element-context))))

will return `paragraph'.


Regards,

-- 
Nicolas Goaziou

Re: Inserting a comma as prefix of headlines (in Org code blocks)

[Bastien]

Hi Nicolas,

thanks for the thorough feedback.  

I reverted the change I made.
I need to think more about the issue.

Best,

-- 
 Bastien

Re: Inserting a comma as prefix of headlines (in Org code blocks)

[Nicolas Goaziou]

Hello,

Bastien <bzg@altern.org> writes:

> thanks for the thorough feedback.  
>
> I reverted the change I made.

Thank you.

> I need to think more about the issue.

I can write a document describing Org syntax, as seen by the parser, if
needed. It may serve as a first draft for an appendix in the Org manual.


Regards,

-- 
Nicolas Goaziou

Re: Inserting a comma as prefix of headlines (in Org code blocks)

[Bastien]

Hi Nicolas,

Nicolas Goaziou <n.goaziou@gmail.com> writes:

> I can write a document describing Org syntax, as seen by the parser, if
> needed. It may serve as a first draft for an appendix in the Org manual.

It will be useful in general; it will spare us miscommunication on
various aspects of the parser.  So please feel free to go ahead.

But be prepared for dealing with some stubbornness on my side:
documenting the parser does not mean every issue should be solved
thinking in terms of the parser.  Sometimes there should be a
tradeoff between what the parser can parse and what the UI should
offer.

More on this later on, probably tomorrow!

Thanks in any case,

-- 
 Bastien

Re: Inserting a comma as prefix of headlines (in Org code blocks)

[Nicolas Goaziou]

Bastien <bzg@altern.org> writes:

> But be prepared for dealing with some stubbornness on my side:
> documenting the parser does not mean every issue should be solved
> thinking in terms of the parser.  Sometimes there should be a
> tradeoff between what the parser can parse and what the UI should
> offer.

Obviously, it depends on the tradeoff. For example, `org-fill-paragraph'
contains one. Here is an excerpt of its docstring:

  For convenience, when point is at a plain list, an item or a footnote
  definition, try to fill the first paragraph within.

This is perfectly fine because it doesn't go against the parser. It
merely acts as if the point was elsewhere in the buffer. But it doesn't
pretend that the buffer is different.

Another possible kind of tradeoff is commands working on a region.
Regions don't mean anything for the parser, so commands acting of them
may ignore what the parser knows about the buffer.

Nevertheless, in other cases, I highly suggest to never discard the
output from the parser. A command ought to always consider that the
parser is correct. So, if it means to go against it, it should be
specified explicitly somewhere (e.g. C-S-up can clearly break the
syntax).

Commands activated through simple keys (i.e. TAB) on basic syntax (i.e.
an headline) shouldn't, IMO, belong to that category, as it would
confuse users even more.


Regards,

-- 
Nicolas Goaziou

Re: Inserting a comma as prefix of headlines (in Org code blocks)

[Achim Gratz]

Nicolas Goaziou <n.goaziou <at> gmail.com> writes:
> I can write a document describing Org syntax, as seen by the parser, if
> needed. It may serve as a first draft for an appendix in the Org manual.

This would be very welcome.  If you think I can be of help with any of this,
please let me know.


Regards,
Achim.

Re: Inserting a comma as prefix of headlines (in Org code blocks)

[François Pinard]

Nicolas Goaziou <n.goaziou@gmail.com> writes:

> I can write a document describing Org syntax, as seen by the parser [...]

That would undoubtedly be useful.

There is a need I often have, and never found the time to fill so far,
for a dependable Python parser for Org syntax.  I thought I could read
the Emacs Lisp code of the parser to get a firm grasp of the syntax
first, but surely, a formal document acting as a reference might be
easier for me to get me started, at least in that it separates what is
official and what is incidental in the syntax.  I got the vague
impression (unsubstantiated right now, I've not been in that area of Org
for a while) that it would help uncoupling Org syntax from some Emacs
intricacies.  I'd need a serious revision to get a firmer opinion.

François

P.S. Another need I think I'll soon have is some machinery for
generating HTML out of dynamic Org snippets, using Python.  I could of
course have Emacs running in background to "serve" a Web application,
but it looks rather awkward to me, and for one specific application of
ours, I really starve for using Org instead of HTML, whenever possible.

Re: Inserting a comma as prefix of headlines (in Org code blocks)

[Yagnesh Raghava Yakkala]

On Mar 01 2013, François Pinard <pinard@iro.umontreal.ca> wrote:

> Nicolas Goaziou <n.goaziou@gmail.com> writes:
>
>> I can write a document describing Org syntax, as seen by the parser [...]
>
> That would undoubtedly be useful.
>
> There is a need I often have, and never found the time to fill so far,
> for a dependable Python parser for Org syntax.  

Not sure how complete it is here is one project., https://github.com/tkf/orgparse 

-- 
ఎందరో మహానుభావులు అందరికి వందనములు.
YYR

Re: Inserting a comma as prefix of headlines (in Org code blocks)

[François Pinard]

Yagnesh Raghava Yakkala <hi@yagnesh.org> writes:

>> There is a need I often have, and never found the time to fill so far,
>> for a dependable Python parser for Org syntax.  

> Not sure how complete it is [...]  https://github.com/tkf/orgparse

Thanks for the pointer.  With about 150 commits already, it seems that a
good effort was invested in it.  Do you know happen to know how
conforming it is?

I wrote many ad hoc parsers for Org already, but what I would like
is something really close to the parser which comes with the new
exporter, both in syntax and concept nomenclature, so I could switch to
it once and for all and delete all my hacks. :-)

François

Re: Inserting a comma as prefix of headlines (in Org code blocks)

[Yagnesh Raghava Yakkala]

[CC'ed to Takafumi Arakaki, author of orgparse]

Hello François,

> Do you know happen to know how conforming it is?

I can't comment on that, since I haven't really used it for anything.

> I wrote many ad hoc parsers for Org already, but what I would like
> is something really close to the parser which comes with the new
> exporter, both in syntax and concept nomenclature, 

AFAIU it is pretty cleanly written and even includes tests., though its just a
reader and can't be used for exporting right now. By glancing through It seems
it doesn't entirely know Org's syntax, especially babel related.

tkf may tell us more.

Thanks.,
-- 
ఎందరో మహానుభావులు అందరికి వందనములు.
YYR

Re: Inserting a comma as prefix of headlines (in Org code blocks)

[Nicolas Goaziou]

Achim Gratz <Stromeko@NexGo.DE> writes:

> Nicolas Goaziou <n.goaziou <at> gmail.com> writes:
>> I can write a document describing Org syntax, as seen by the parser, if
>> needed. It may serve as a first draft for an appendix in the Org manual.
>
> This would be very welcome.  If you think I can be of help with any of this,
> please let me know.

Thank you for your offer. I appreciate it.

To begin with, I am going to produce a terse initial document, on my
own. I'll post it on the ML and then, we will able to discuss what can
be improved, and how. At this point I'll probably need some help.


Regards,

-- 
Nicolas Goaziou

Re: Inserting a comma as prefix of headlines (in Org code blocks)

[Takafumi Arakaki]

Hi,

Indeed, orgparse can parse only subset of org syntax.  It is primary
developed for OrgViz [1] which is an app to view your org files in
various formats including calendar, graph and table.  That's why it
doesn't understand babel stuff.  I thought I'd advertise it in org
ML at some point but I was too lazy to do that.

As orgparse supports `:PROPERTIES:', I think it is not hard to support
babel syntax.  If somebody can write a patch I am happy to review and
pull it.

[1] https://github.com/tkf/orgviz


On Thu, Feb 28, 2013 at 9:06 PM, Yagnesh Raghava Yakkala <hi@yagnesh.org> wrote:
>
> [CC'ed to Takafumi Arakaki, author of orgparse]
>
> Hello François,
>
>> Do you know happen to know how conforming it is?
>
> I can't comment on that, since I haven't really used it for anything.
>
>> I wrote many ad hoc parsers for Org already, but what I would like
>> is something really close to the parser which comes with the new
>> exporter, both in syntax and concept nomenclature,
>
> AFAIU it is pretty cleanly written and even includes tests., though its just a
> reader and can't be used for exporting right now. By glancing through It seems
> it doesn't entirely know Org's syntax, especially babel related.
>
> tkf may tell us more.
>
> Thanks.,
> --
> ఎందరో మహానుభావులు అందరికి వందనములు.
> YYR