[RFC] Org Minor Mode?

[RFC] Org Minor Mode?

[Thorsten Jolitz]

Hi List, 

I cite from a recent thread on Emacs Help:

#+begin_quote
Right, I'm very much convinced that the power of Org-mode could be
unleashed in a true org-minor-mode that works in the comment-sections of
programming major-modes.

[...]

The major "flaw" of Org-mode that inhibits its use as minor-mode is the
wide-spread use of hard-coded regexps, i.e. regexps of this form

,--------------
| "^\\*+ ... $"
`--------------

in many many variants all over the place. Those three elements "^"
(bol), "$" (eol) and "*" (star) are not portable, since in
comment-sections of programming major-modes the above should look
like:

PicoLisp  (comment-start repeated with padding)
,--------------
| "^## *+ ... $" 
`--------------

Elisp outshine-style (comment-start repeated with padding)
,--------------
| "^;; *+ ... $" 
`--------------

old-school Elisp (comment-start repeated no padding, different star)
,--------------
| "^;;;+ ... $" 
`--------------
#+end_quote

Now I just announced [[https://github.com/tj64/drx][drx.el]] on Emacs
Help, I cite again:

#+begin_quote
Its main purpose is to abstract from hardcoded  "^", "$" and "\\*" in
regexps. Here are a few examples:

# basic usage
#+begin_src emacs-lisp
 (drx "foo" t nil t)
#+end_src

#+results:
: ^foo$

# org-mode version
#+begin_src emacs-lisp
 (drx " foo" t '(t "+") t)
#+end_src

#+results:
: ^\(\*+\) foo$

# css-mode version with # as star 
#+begin_src emacs-lisp
  (let ((drx-BOL "^/* ")
        (drx-EOL "*/$")
        (drx-STAR "#"))
    (drx " foo" t '(t "+") t))
#+end_src

#+results:
: ^/* \(#+\) foo*/$

Many more things are possible, the more than 80 ERT tests in drx.el give
many examples with expected output.
#+end_quote

What do you think - is there any chance that Org-mode switches from
static hardcoded regexp strings (all over the place) to dynamic
regexps calculated at runtime (using libraries like drx.el or rx.el)?

There would be a cost definitely, but then a true org-minor-mode could
become reality that offers the full power of Org-mode in programming
mode's comment section (I'm actually working on it by merging outshine
with orgstruct: [[https://github.com/tj64/omm][omm.el]]).

-- 
cheers,
Thorsten

Re: [RFC] Org Minor Mode?

[Nicolas Goaziou]

Hello,

Thorsten Jolitz <tjolitz@gmail.com> writes:

> What do you think - is there any chance that Org-mode switches from
> static hardcoded regexp strings (all over the place) to dynamic
> regexps calculated at runtime (using libraries like drx.el or rx.el)?

I hope not. The syntax should stabilize, not drift away.

> There would be a cost definitely, but then a true org-minor-mode could
> become reality that offers the full power of Org-mode in programming
> mode's comment section (I'm actually working on it by merging outshine
> with orgstruct: [[https://github.com/tj64/omm][omm.el]]).

I don't see why you would need the "full power of Org-mode" (whatever
that means) in mere comments. In any case, you can override
`org-outline-regexp' and `org-outline-regexp-bol' in any minor mode.


Regards,

-- 
Nicolas Goaziou

Re: [RFC] Org Minor Mode?

[Bastien]

Hi Thorsten,

Nicolas Goaziou <n.goaziou@gmail.com> writes:

> Thorsten Jolitz <tjolitz@gmail.com> writes:
>
>> What do you think - is there any chance that Org-mode switches from
>> static hardcoded regexp strings (all over the place) to dynamic
>> regexps calculated at runtime (using libraries like drx.el or rx.el)?
>
> I hope not. The syntax should stabilize, not drift away.

Agreed.  Maybe there are some hardcoded regexps that we can factor
out, but dynamically building those fundamental regexp is a deadend.

Best,

-- 
 Bastien

Re: [RFC] Org Minor Mode?

[Richard Lawrence]

Hi Thorsten,

Bastien <bzg@gnu.org> writes:

> Nicolas Goaziou <n.goaziou@gmail.com> writes:
>
>> Thorsten Jolitz <tjolitz@gmail.com> writes:
>>
>>> What do you think - is there any chance that Org-mode switches from
>>> static hardcoded regexp strings (all over the place) to dynamic
>>> regexps calculated at runtime (using libraries like drx.el or rx.el)?
>>
>> I hope not. The syntax should stabilize, not drift away.
>
> Agreed.  Maybe there are some hardcoded regexps that we can factor
> out, but dynamically building those fundamental regexp is a deadend.

I agree with what Nicolas and Bastien have said, but I wanted to say
that I think there is an interesting idea in Thorsten's post that
shouldn't be dismissed out of hand.

Org provides a set of UI concepts (tree-like structure, visibility
cycling, tree filtering, task state tracking, building an agenda from
multiple sources, ...)  that map nicely onto a lot of other situations,
and would be really handy to have access to even when the syntax of the
underlying file is incompatible with Org's syntax.

There are two ways to think about Org syntax, which I think should be
distinguished here.  One is as the grammar of a .org file: basically, a
set of rules that allow a sequence of characters to be parsed into an
AST.  The other way to think about Org syntax is the way Lisp
programmers sometimes talk about syntax: as the AST itself, the
collection of Lisp data types and their interrelationships that define a
valid Org document.

If Org were to evolve to the point where the UI concepts were
implemented purely as transformations on an AST -- on Org syntax in the
second sense -- then the way would be clear for making those concepts
available in editing modes where the grammar of the underlying file is
incompatible with Org syntax in the first sense.  A programming mode
could, say, parse comments into an Org AST, then expose that AST to the
functions implementing Org's UI concepts.  Et voila: you get visibility
cycling, task state tracking, agendas...in your source code comments.

One sort of use case where I think this idea could really shine is in
dealing with email.  Obviously, the grammar of the underlying mail files
(say, in a Maildir) will never be compatible with Org syntax in the
first sense.  But Org handles so many of the concepts that apply to
email (threading messages into hierarchies, visibility cycling, tagging,
sorting by date or priority, thinking of messages as tasks to be dealt
with, dealing with attachments) in such a nice way that I find myself
sorely missing Org whenever I read mail in a client that doesn't
implement them as nicely -- which is all of them.  If it were possible
to build a parser for message files that transformed them into an Org
AST, the mail client of my dreams would be in reach.

I have no idea if evolving Org in this direction is feasible or even
really desireable.  It may be that the two notions of Org syntax are
tightly coupled in principle, so that the idea of producing an Org AST
from an alternative underlying file format will never make sense.  But I
think that would be surprising.  

This evolution would clearly require more work than just abstracting out
the regular expressions that implement much of Org's syntax in the first
sense, and I think Bastien and Nicolas are right that we don't want
either notion of Org syntax to become less stable.  Still, I think
there's a lot of interesting possibilities we could explore if Org's
implementations of the two notions of syntax were to become less tightly
coupled.

Best,
Richard

Re: [RFC] Org Minor Mode?

[Thorsten Jolitz]

Richard Lawrence <richard.lawrence@berkeley.edu> writes:

Hi Richard,

> Bastien <bzg@gnu.org> writes:
>
>> Nicolas Goaziou <n.goaziou@gmail.com> writes:
>>
>>> Thorsten Jolitz <tjolitz@gmail.com> writes:
>>>
>>>> What do you think - is there any chance that Org-mode switches from
>>>> static hardcoded regexp strings (all over the place) to dynamic
>>>> regexps calculated at runtime (using libraries like drx.el or rx.el)?
>>>
>>> I hope not. The syntax should stabilize, not drift away.
>>
>> Agreed.  Maybe there are some hardcoded regexps that we can factor
>> out, but dynamically building those fundamental regexp is a deadend.
>
> I agree with what Nicolas and Bastien have said, but I wanted to say
> that I think there is an interesting idea in Thorsten's post that
> shouldn't be dismissed out of hand.
>
> Org provides a set of UI concepts (tree-like structure, visibility
> cycling, tree filtering, task state tracking, building an agenda from
> multiple sources, ...)  that map nicely onto a lot of other situations,
> and would be really handy to have access to even when the syntax of the
> underlying file is incompatible with Org's syntax.
>
> There are two ways to think about Org syntax, which I think should be
> distinguished here.  One is as the grammar of a .org file: basically, a
> set of rules that allow a sequence of characters to be parsed into an
> AST.  The other way to think about Org syntax is the way Lisp
> programmers sometimes talk about syntax: as the AST itself, the
> collection of Lisp data types and their interrelationships that define a
> valid Org document.
>
> If Org were to evolve to the point where the UI concepts were
> implemented purely as transformations on an AST -- on Org syntax in the
> second sense -- then the way would be clear for making those concepts
> available in editing modes where the grammar of the underlying file is
> incompatible with Org syntax in the first sense.  A programming mode
> could, say, parse comments into an Org AST, then expose that AST to the
> functions implementing Org's UI concepts.  Et voila: you get visibility
> cycling, task state tracking, agendas...in your source code comments.
>
> One sort of use case where I think this idea could really shine is in
> dealing with email.  Obviously, the grammar of the underlying mail files
> (say, in a Maildir) will never be compatible with Org syntax in the
> first sense.  But Org handles so many of the concepts that apply to
> email (threading messages into hierarchies, visibility cycling, tagging,
> sorting by date or priority, thinking of messages as tasks to be dealt
> with, dealing with attachments) in such a nice way that I find myself
> sorely missing Org whenever I read mail in a client that doesn't
> implement them as nicely -- which is all of them.  If it were possible
> to build a parser for message files that transformed them into an Org
> AST, the mail client of my dreams would be in reach.
>
> I have no idea if evolving Org in this direction is feasible or even
> really desireable.  It may be that the two notions of Org syntax are
> tightly coupled in principle, so that the idea of producing an Org AST
> from an alternative underlying file format will never make sense.  But I
> think that would be surprising.  
>
> This evolution would clearly require more work than just abstracting out
> the regular expressions that implement much of Org's syntax in the first
> sense, and I think Bastien and Nicolas are right that we don't want
> either notion of Org syntax to become less stable.  Still, I think
> there's a lot of interesting possibilities we could explore if Org's
> implementations of the two notions of syntax were to become less tightly
> coupled.

thanks for your long and interesting comment!

But thats very ambitious, I'm actually looking for something much more
prosaic - being able to structure and use a buffer/file in a programming
mode like an Org-mode buffer/file besides the fact the headlines and
text needs to be outcommented. 

Thats partly possible with orgstruct or outshine, but I think there is
potential for more if Org functions could deal with outcommented
syntax-elements. And at the lowest level its all about regexps that
don't match because there is comment-syntax behind the "^" and before
the "$" (and maybe a different star is used instead of "*"), e.g.

(outshine-style) Elisp
,--------------------
| "^;; \\* Headline$"
`--------------------

oldschool Elisp
,-----------------
| "^;;; Headline$"
`-----------------

or 

CSS
,-----------------------
| "^/* \\* Headline */$"
`-----------------------

instead of the default Org-mode

,-----------------
| "^\\* Headline$"
`-----------------

-- 
cheers,
Thorsten

Re: [RFC] Org Minor Mode?

[Samuel Wales]

another option is to create annotation mechanisms that are so
compelling that you don't need org in non-org files.

your lists and tasks would stay in your org agenda files, but your
external files would be able to show (via overlays) and link to the
annotations in org.  in turn, your annotations in org would be able to
send you to the spot in your extenal files that they refer to.

we have a ton of annotation mechanisms in emacs and org, but they can
perhaps be made more compelling in this way.


On 4/11/14, Richard Lawrence <richard.lawrence@berkeley.edu> wrote:
> Hi Thorsten,
>
> Bastien <bzg@gnu.org> writes:
>
>> Nicolas Goaziou <n.goaziou@gmail.com> writes:
>>
>>> Thorsten Jolitz <tjolitz@gmail.com> writes:
>>>
>>>> What do you think - is there any chance that Org-mode switches from
>>>> static hardcoded regexp strings (all over the place) to dynamic
>>>> regexps calculated at runtime (using libraries like drx.el or rx.el)?
>>>
>>> I hope not. The syntax should stabilize, not drift away.
>>
>> Agreed.  Maybe there are some hardcoded regexps that we can factor
>> out, but dynamically building those fundamental regexp is a deadend.
>
> I agree with what Nicolas and Bastien have said, but I wanted to say
> that I think there is an interesting idea in Thorsten's post that
> shouldn't be dismissed out of hand.
>
> Org provides a set of UI concepts (tree-like structure, visibility
> cycling, tree filtering, task state tracking, building an agenda from
> multiple sources, ...)  that map nicely onto a lot of other situations,
> and would be really handy to have access to even when the syntax of the
> underlying file is incompatible with Org's syntax.
>
> There are two ways to think about Org syntax, which I think should be
> distinguished here.  One is as the grammar of a .org file: basically, a
> set of rules that allow a sequence of characters to be parsed into an
> AST.  The other way to think about Org syntax is the way Lisp
> programmers sometimes talk about syntax: as the AST itself, the
> collection of Lisp data types and their interrelationships that define a
> valid Org document.
>
> If Org were to evolve to the point where the UI concepts were
> implemented purely as transformations on an AST -- on Org syntax in the
> second sense -- then the way would be clear for making those concepts
> available in editing modes where the grammar of the underlying file is
> incompatible with Org syntax in the first sense.  A programming mode
> could, say, parse comments into an Org AST, then expose that AST to the
> functions implementing Org's UI concepts.  Et voila: you get visibility
> cycling, task state tracking, agendas...in your source code comments.
>
> One sort of use case where I think this idea could really shine is in
> dealing with email.  Obviously, the grammar of the underlying mail files
> (say, in a Maildir) will never be compatible with Org syntax in the
> first sense.  But Org handles so many of the concepts that apply to
> email (threading messages into hierarchies, visibility cycling, tagging,
> sorting by date or priority, thinking of messages as tasks to be dealt
> with, dealing with attachments) in such a nice way that I find myself
> sorely missing Org whenever I read mail in a client that doesn't
> implement them as nicely -- which is all of them.  If it were possible
> to build a parser for message files that transformed them into an Org
> AST, the mail client of my dreams would be in reach.
>
> I have no idea if evolving Org in this direction is feasible or even
> really desireable.  It may be that the two notions of Org syntax are
> tightly coupled in principle, so that the idea of producing an Org AST
> from an alternative underlying file format will never make sense.  But I
> think that would be surprising.
>
> This evolution would clearly require more work than just abstracting out
> the regular expressions that implement much of Org's syntax in the first
> sense, and I think Bastien and Nicolas are right that we don't want
> either notion of Org syntax to become less stable.  Still, I think
> there's a lot of interesting possibilities we could explore if Org's
> implementations of the two notions of syntax were to become less tightly
> coupled.
>
> Best,
> Richard
>
>
>


-- 
The Kafka Pandemic: http://thekafkapandemic.blogspot.com

The disease DOES progress.  MANY people have died from it.  And
ANYBODY can get it.

Denmark: free Karina Hansen NOW.

Re: [RFC] Org Minor Mode?

[Thorsten Jolitz]

Samuel Wales <samologist@gmail.com> writes:

Hi Samuel, 

> another option is to create annotation mechanisms that are so
> compelling that you don't need org in non-org files.
>
> your lists and tasks would stay in your org agenda files, but your
> external files would be able to show (via overlays) and link to the
> annotations in org.  in turn, your annotations in org would be able to
> send you to the spot in your extenal files that they refer to.
>
> we have a ton of annotation mechanisms in emacs and org, but they can
> perhaps be made more compelling in this way.

I think there probably do exist quite a lot of individual customized
systems to connect Org planning files to associated source code files,
and this is definitely a viable use pattern. 

But I think a simpler more direct approach is possible - simply an
outline-minor-mode on steroids (=> org-minor-mode). If I structure my
source file with headlines like an org file, do visibility-cycling,
structure editing etc. like in Org-mode - why not add TODO's and
priorities and tags and properties to my headlines? The functionality to
do that is already there in Org-mode, and the syntax-elements used would
be exactly the same. Except the comment-syntax involved ...

> On 4/11/14, Richard Lawrence <richard.lawrence@berkeley.edu> wrote:
>> Hi Thorsten,
>>
>> Bastien <bzg@gnu.org> writes:
>>
>>> Nicolas Goaziou <n.goaziou@gmail.com> writes:
>>>
>>>> Thorsten Jolitz <tjolitz@gmail.com> writes:
>>>>
>>>>> What do you think - is there any chance that Org-mode switches from
>>>>> static hardcoded regexp strings (all over the place) to dynamic
>>>>> regexps calculated at runtime (using libraries like drx.el or rx.el)?
>>>>
>>>> I hope not. The syntax should stabilize, not drift away.
>>>
>>> Agreed.  Maybe there are some hardcoded regexps that we can factor
>>> out, but dynamically building those fundamental regexp is a deadend.
>>
>> I agree with what Nicolas and Bastien have said, but I wanted to say
>> that I think there is an interesting idea in Thorsten's post that
>> shouldn't be dismissed out of hand.
>>
>> Org provides a set of UI concepts (tree-like structure, visibility
>> cycling, tree filtering, task state tracking, building an agenda from
>> multiple sources, ...)  that map nicely onto a lot of other situations,
>> and would be really handy to have access to even when the syntax of the
>> underlying file is incompatible with Org's syntax.
>>
>> There are two ways to think about Org syntax, which I think should be
>> distinguished here.  One is as the grammar of a .org file: basically, a
>> set of rules that allow a sequence of characters to be parsed into an
>> AST.  The other way to think about Org syntax is the way Lisp
>> programmers sometimes talk about syntax: as the AST itself, the
>> collection of Lisp data types and their interrelationships that define a
>> valid Org document.
>>
>> If Org were to evolve to the point where the UI concepts were
>> implemented purely as transformations on an AST -- on Org syntax in the
>> second sense -- then the way would be clear for making those concepts
>> available in editing modes where the grammar of the underlying file is
>> incompatible with Org syntax in the first sense.  A programming mode
>> could, say, parse comments into an Org AST, then expose that AST to the
>> functions implementing Org's UI concepts.  Et voila: you get visibility
>> cycling, task state tracking, agendas...in your source code comments.
>>
>> One sort of use case where I think this idea could really shine is in
>> dealing with email.  Obviously, the grammar of the underlying mail files
>> (say, in a Maildir) will never be compatible with Org syntax in the
>> first sense.  But Org handles so many of the concepts that apply to
>> email (threading messages into hierarchies, visibility cycling, tagging,
>> sorting by date or priority, thinking of messages as tasks to be dealt
>> with, dealing with attachments) in such a nice way that I find myself
>> sorely missing Org whenever I read mail in a client that doesn't
>> implement them as nicely -- which is all of them.  If it were possible
>> to build a parser for message files that transformed them into an Org
>> AST, the mail client of my dreams would be in reach.
>>
>> I have no idea if evolving Org in this direction is feasible or even
>> really desireable.  It may be that the two notions of Org syntax are
>> tightly coupled in principle, so that the idea of producing an Org AST
>> from an alternative underlying file format will never make sense.  But I
>> think that would be surprising.
>>
>> This evolution would clearly require more work than just abstracting out
>> the regular expressions that implement much of Org's syntax in the first
>> sense, and I think Bastien and Nicolas are right that we don't want
>> either notion of Org syntax to become less stable.  Still, I think
>> there's a lot of interesting possibilities we could explore if Org's
>> implementations of the two notions of syntax were to become less tightly
>> coupled.
>>
>> Best,
>> Richard
>>
>>
>>

-- 
cheers,
Thorsten

Re: [RFC] Org Minor Mode?

[Thorsten Jolitz]

Bastien <bzg@gnu.org> writes:

> Hi Thorsten,
>
> Nicolas Goaziou <n.goaziou@gmail.com> writes:
>
>> Thorsten Jolitz <tjolitz@gmail.com> writes:
>>
>>> What do you think - is there any chance that Org-mode switches from
>>> static hardcoded regexp strings (all over the place) to dynamic
>>> regexps calculated at runtime (using libraries like drx.el or rx.el)?
>>
>> I hope not. The syntax should stabilize, not drift away.
>
> Agreed.  Maybe there are some hardcoded regexps that we can factor
> out, but dynamically building those fundamental regexp is a deadend.

In the case of regexps its of course not considered bad practice to
write "^foo$" many times in a program, so nobody is to blame here, but
the issue is the same as writing (message "What a wonderful world") many
times in a program and then entering a depressed state.

Would have been better to write (message "What a %s world"
current-world-view) ...

-- 
cheers,
Thorsten

Re: [RFC] Org Minor Mode?

[Bastien]

Thorsten Jolitz <tjolitz@gmail.com> writes:

> In the case of regexps its of course not considered bad practice to
> write "^foo$" many times in a program, so nobody is to blame here, but
> the issue is the same as writing (message "What a wonderful world") many
> times in a program and then entering a depressed state.
>
> Would have been better to write (message "What a %s world"
> current-world-view) ...

... except when the world is *always* wonderful!

-- 
 Bastien

Re: [RFC] Org Minor Mode?

[Thorsten Jolitz]

Nicolas Goaziou <n.goaziou@gmail.com> writes:

> Hello,
>
> Thorsten Jolitz <tjolitz@gmail.com> writes:
>
>> What do you think - is there any chance that Org-mode switches from
>> static hardcoded regexp strings (all over the place) to dynamic
>> regexps calculated at runtime (using libraries like drx.el or rx.el)?
>
> I hope not. The syntax should stabilize, not drift away.

This is not so much about changing the syntax but rather about the
common programming practice to define and change/modify a variable or
constant in one place only. 

#+begin_src emacs-lisp
(defvar BOL "^") ; once 
(looking-at (concat BOL "foo")) ; many times
#+end_src

instead of

#+begin_src emacs-lisp
(looking-at "^foo") ; many times
#+end_src

The need to abstract from the Org star '*' too would only arise from
need to deal with Emacs Lisp coding conventions (;;; Level1 Headline),
which is a (unfortunately very important) special case. In all other
cases it would be only about BOL ("^") and EOL ("$") in regexps.

>> There would be a cost definitely, but then a true org-minor-mode could
>> become reality that offers the full power of Org-mode in programming
>> mode's comment section (I'm actually working on it by merging outshine
>> with orgstruct: [[https://github.com/tj64/omm][omm.el]]).

> I don't see why you would need the "full power of Org-mode" (whatever
> that means) in mere comments. 

That means that Org headlines are much more 'intelligent' than outline
headlines (they know about todos, priorities, tags, timestamps,
properties, planning, clocking, being archived etc.) and one could
have that kind of intelligent headlines in other modes too with a true
org-minor-mode.

-- 
cheers,
Thorsten

Re: [RFC] Org Minor Mode?

[Bastien]

Thorsten Jolitz <tjolitz@gmail.com> writes:

> That means that Org headlines are much more 'intelligent' than outline
> headlines (they know about todos, priorities, tags, timestamps,
> properties, planning, clocking, being archived etc.) and one could
> have that kind of intelligent headlines in other modes too with a true
> org-minor-mode.

Joke aside, did you closely look at orgstruct-mode?
I guess so if you started outorg.el.

My point is: I'd rather improve orgstruct-mode than rethink
an org-minor-mode from scratch.  But that's just me of course.

-- 
 Bastien

Re: [RFC] Org Minor Mode?

[Thorsten Jolitz]

Bastien <bzg@gnu.org> writes:

> Thorsten Jolitz <tjolitz@gmail.com> writes:
>
>> That means that Org headlines are much more 'intelligent' than outline
>> headlines (they know about todos, priorities, tags, timestamps,
>> properties, planning, clocking, being archived etc.) and one could
>> have that kind of intelligent headlines in other modes too with a true
>> org-minor-mode.
>
> Joke aside, did you closely look at orgstruct-mode?
> I guess so if you started outorg.el.

Yes! Actually, omm.el is a merge of outshine.el and the orgstruct
section from org.el, i.e. the entire orgstruct code has been copied to
omm.el and therefore could be removed from org.el in case org-minor-mode
ever becomes reality (and omm.el part of org-mode). 

I think org(struct)-style list and table editing in other text-modes is
a kind of special case for an org-minor-mode, though a very useful
one. The default case would be its (mode agnostic) use in programming
modes and with outcommented text. In that case outshine and
outorg/poporg offer some useful ideas and functionality too.

> My point is: I'd rather improve orgstruct-mode than rethink
> an org-minor-mode from scratch.  But that's just me of course.

I think outorg and poporg are quite heavy competitors to
org(struct)-style list and table editing, since they offer the full
power of Org-mode, work in comment sections and docstrings (too), and
avoid possible mode conflicts by simply reverting the
`org-edit-special' idea => work in a programming mode and launch
temporary org-mode edit buffers for occasional text editing.

So for text/list/table editing I think there could be user options for
deciding between orgstruct and/or outorg&poporg, so that none of the
existing efforts is lost. The real value and innovation of a true
org-minor-mode would be to introduce Org's intelligent headlines and all
the related functionality into the world of outcommented text in
programming modes.

-- 
cheers,
Thorsten

Re: [RFC] Org Minor Mode?

[Bastien]

Hi Thorsten,

Thorsten Jolitz <tjolitz@gmail.com> writes:

> The real value and innovation of a true
> org-minor-mode would be to introduce Org's intelligent headlines and all
> the related functionality into the world of outcommented text in
> programming modes.

Yes.  We could have `orgstruct-comment-prefix-regexp' along with
`orgstruct-heading-prefix-regexp'.  I'll check this, thanks for
the idea!

-- 
 Bastien

Re: [RFC] Org Minor Mode?

[Thorsten Jolitz]

Bastien <bzg@gnu.org> writes:

Hi Bastien,

> Thorsten Jolitz <tjolitz@gmail.com> writes:
>
>> The real value and innovation of a true
>> org-minor-mode would be to introduce Org's intelligent headlines and all
>> the related functionality into the world of outcommented text in
>> programming modes.
>
> Yes.  We could have `orgstruct-comment-prefix-regexp' along with
> `orgstruct-heading-prefix-regexp'.  I'll check this, thanks for
> the idea!

As long as there are hardcoded regexp searches all over the place in
org.el, org-agenda.el, orgtbl.el, and to a minor extends almost all
other Org libraries, I don't think just another outline-regexp can do
the trick - the called functions will fail when trying to match

,--------
| "^foo$"
`--------

but looking at

,---------------------------------------------------------------
| "^<<comment-starter-syntax>>foo<<maybe-comment-end-syntax>>$"
`---------------------------------------------------------------

Furthermore calling Org functions modifies the buffer, sometimes
inserting new lines, sometimes not. An Org minor-mode would need to
detect all newly inserted lines after a command call and outcomment them
with the programming major-mode's comment syntax. 

I already found a way to do that using external calls to diff, its
implemented in omm.el (needs to be debugged though, I stopped working on
it because it seemed the idea of an Org Minor Mode was a bit unpopular). 

In summary, its about:

 1. generalize the regexp constants and vars (allow for comment-syntax,
 when org-minor-mode)

 2. deal with hardcoded regexp-snippets in functions (my proposoal:
 replace "^" with org-BOL, "$" with org-EOL, "\\*" with org-STAR)

 3. outcomment new lines after calls to Org commands. 

All the other stuff should be already there in orgstruct and outshine. 

-- 
cheers,
Thorsten

Re: [RFC] Org Minor Mode?

[Bastien]

Hi Thorsten,

Thorsten Jolitz <tjolitz@gmail.com> writes:

> In summary, its about:
>
>  1. generalize the regexp constants and vars (allow for comment-syntax,
>  when org-minor-mode)
>
>  2. deal with hardcoded regexp-snippets in functions (my proposoal:
>  replace "^" with org-BOL, "$" with org-EOL, "\\*" with org-STAR)
>
>  3. outcomment new lines after calls to Org commands.

I still think there is a simpler way, I'll explore this and
I'll let you know.

-- 
 Bastien

Re: [RFC] Org Minor Mode?

[Ilya Shlyakhter]

On 4/19/2014 8:57 AM, Bastien wrote:
> Hi Thorsten,
>
> Thorsten Jolitz <tjolitz@gmail.com> writes:
>
>> In summary, its about:
>>
>>   1. generalize the regexp constants and vars (allow for comment-syntax,
>>   when org-minor-mode)
>>
>>   2. deal with hardcoded regexp-snippets in functions (my proposoal:
>>   replace "^" with org-BOL, "$" with org-EOL, "\\*" with org-STAR)
>>
>>   3. outcomment new lines after calls to Org commands.
>
> I still think there is a simpler way, I'll explore this and
> I'll let you know.
>

What about using advice on regexp functions to transform the regexps
(when invoked in org-minor-mode buffers) so that $ is replaced with "; 
$" etc?

One other issue is the use of the number of stars to determine outline 
level.

Re: [RFC] Org Minor Mode?

[Thorsten Jolitz]

Ilya Shlyakhter <ilya_shl@alum.mit.edu> writes:

> On 4/19/2014 8:57 AM, Bastien wrote:
>> Hi Thorsten,
>>
>> Thorsten Jolitz <tjolitz@gmail.com> writes:
>>
>>> In summary, its about:
>>>
>>>   1. generalize the regexp constants and vars (allow for comment-syntax,
>>>   when org-minor-mode)
>>>
>>>   2. deal with hardcoded regexp-snippets in functions (my proposoal:
>>>   replace "^" with org-BOL, "$" with org-EOL, "\\*" with org-STAR)
>>>
>>>   3. outcomment new lines after calls to Org commands.
>>
>> I still think there is a simpler way, I'll explore this and
>> I'll let you know.
>>
>
> What about using advice on regexp functions to transform the regexps
> (when invoked in org-minor-mode buffers) so that $ is replaced with ";
> $" etc?

Its definitely one option, but its always said in the manuals that, if
possible, one should rather modify the sources than using advices, and
since Org-mode is not abandon-ware at all, I thought I ask first how
chances are for such modifications. 

> One other issue is the use of the number of stars to determine outline
> level.

Since outshine should bring the look&feel of Org-mode to programming
language buffers, using other language elements (like e.g. functions) as
outline items is not only a technical problem, but a conceptual problem
too. In Org-mode, functions are stored in src-blocks, they are not
headlines. Thus they should be treated in outshine rather like plain
lists or drawers or other elements that change visibility but are no
headline. 

-- 
cheers,
Thorsten

Re: [RFC] Org Minor Mode?

[Bastien]

Ilya Shlyakhter <ilya_shl@alum.mit.edu> writes:

> What about using advice on regexp functions to transform the regexps
> (when invoked in org-minor-mode buffers) so that $ is replaced with ";
> $" etc?

We should absolutely avoid advice in code.

-- 
 Bastien

Re: [RFC] Org Minor Mode?

[Ilya Shlyakhter]

Bastien <bzg@gnu.org> writes:
> We should absolutely avoid advice in code.

Fully agree.  (I was thinking of using an flet-like construct to
temporarily rebind functions for the duration of calls, rather than
permanent advice -- see elu-flet in
https://github.com/notestaff/elu/blob/master/elu.el -- but, still agree.)

What would be the downside of abstracting away the headline syntax in
the Org code?  (Other than the admittedly significant issue of
disturbing well-tested core code.  On the other hand, abstracted code
would be more readable & maintainable; see e.g. Org syntax defined with
an rx-like extension:
https://github.com/notestaff/elu/blob/master/rxx-org.el , or with
Thorsten's drx.el )

Obviously this would be a lot of work, but the upside of having a full
Org minor mode (in particular for outshine-type use) would make it worth
it, for me at least.

Re: [RFC] Org Minor Mode?

[Bastien]

Hi Ilya,

Ilya Shlyakhter <ilya239@gmail.com> writes:

> What would be the downside of abstracting away the headline syntax in
> the Org code?

I expect some slow down on core functions -- but I'm not sure.

-- 
 Bastien

Re: [RFC] Org Minor Mode?

[Ilya Shlyakhter]

On 4/10/2014 3:19 PM, Nicolas Goaziou wrote:
> I don't see why you would need the "full power of Org-mode" (whatever
> that means) in mere comments.

There are actually many uses, especially if it becomes possible to treat 
language elements (functions, classes etc) as outline elements
(cf. 
https://github.com/notestaff/outshine/blob/outshine-lang/outshine-lang.el ). 
  Sparse trees and agenda views could be used to find all code elements 
related to a particular aspect of functionality, if items
related to that aspect are tagged with a tag.  Sparse trees could show
just the public (interface) elements.  Basically, outshine with the full 
power of Org and the ability to treat language elements as outline 
headings would add up to the first literate programming system I'd call 
usable.

Re: [RFC] Org Minor Mode?

[Thorsten Jolitz]

Ilya Shlyakhter <ilya_shl@alum.mit.edu> writes:

Hi Ilya,

> On 4/10/2014 3:19 PM, Nicolas Goaziou wrote:
>> I don't see why you would need the "full power of Org-mode" (whatever
>> that means) in mere comments.
>
> There are actually many uses, especially if it becomes possible to
> treat language elements (functions, classes etc) as outline elements
> (cf. https://github.com/notestaff/outshine/blob/outshine-lang/outshine-lang.el
> ). Sparse trees and agenda views could be used to find all code
> elements related to a particular aspect of functionality, if items
> related to that aspect are tagged with a tag.  Sparse trees could show
> just the public (interface) elements.  Basically, outshine with the
> full power of Org and the ability to treat language elements as
> outline headings would add up to the first literate programming system
> I'd call usable.

I think outshine (or a true org-minor-mode) definitely has its merits
when its more about "literate PROGRAMMING" rather than "LITERATE
programming".

I definitely liked your idea to use navi-mode regexps as additional
outline elements - they can easily be defined by the users via
customizable vars `navi-key-mappings' and `navi-keywords', cover a wide
range of syntax elements, and would be defined once but used twice
then. The most obvious choice for outlining would be the regexp stored
in :FUN and called with 'f' in navi-mode, e.g. in elisp:

#+begin_src emacs-lisp
(defcustom navi-key-mappings
  '(("emacs-lisp" . ((:FUN . "f") ...)))
#+end_src

#+begin_src emacs-lisp
(defcustom navi-keywords
  '(("emacs-lisp" . ((:FUN
                       . "^[[:space:]]*(def[maus][^elt][a-z]*\\*? ") ...)))
#+end_src

That way somebody could define a set of predefined keyword searches for a
programming language that is included in navi-mode.el (like I did for
Emacs Lisp, PicoLisp, Org-mode and ESS/R) and users could modify it or
add their own regexps and keybindings. Some of these regexps could
then be reused in outshine for visiility cycling and other stuff. 

There are many Org features I would really like to have in outshine
too, but I realized that reimplementing Org stuff in outshine
would be kind of an uphill battle in the long run, so I would rather
take what is there (orgstruct, outshine, outorg/poporg, navi) and
merge it into a true org-minor-mode reusing existing Org-mode
functionality.

I have a pretty clear picture what is needed for an org-minor-mode,
and started implementation in 

,----------------------------
| https://github.com/tj64/omm
`----------------------------

already. But I feel that this should be done together with the Org
maintainers and the Org community, otherwise bad things could happen:

 - I write the whole thing (changing Org sources) and the maintainers
   say NO to it.

 - I write the whole thing (changing Org sources), the maintainers say
   NO to it, but I want it anyway and become the unvoluntary maintainer
   of an unmaintainable Org fork (nightmare).

 - I write the whole thing w/o changing Org sources (if that is
   possible), e.g. using advices, the Org people like it and reimplement
   it w/o advices, and my effort is wasted.

So I would rather have a clear picture of the 'political situation' wrt
to org-minor-mode and some technical discussion first. But anyway, I
will not have time to work on this again before June.

-- 
cheers,
Thorsten

Re: [RFC] Org Minor Mode?

[Bastien]

Hi Thorsten,

Thorsten Jolitz <tjolitz@gmail.com> writes:

> So I would rather have a clear picture of the 'political situation' wrt
> to org-minor-mode and some technical discussion first. But anyway, I
> will not have time to work on this again before June.

I for one need to have a clearer picture of what such a minor mode
would really do, without getting prematurily lost in the details of
possible implementations.

Once we have a clear goal, let's build a small list of steps, then
submit a patch for the first step.  (We can then work on a separate
branch for the whole feature, and spend months on this branch.)

If the change is relying on too many external libraries, it is not
very likely to happen.

The feature seems to bring interest, but let's refrain from spending
our enthusiasm in early and lengthy discussions that nobody can fully
follow, especially if you won't have time to hack something before
June...

So I really don't want to play the cold fish here, but let's move
with code and small steps rather than discussions and big goals.

-- 
 Bastien

Re: [RFC] Org Minor Mode?

[Thorsten Jolitz]

Bastien <bzg@gnu.org> writes:

Hi Bastien,

> Thorsten Jolitz <tjolitz@gmail.com> writes:
>
>> So I would rather have a clear picture of the 'political situation' wrt
>> to org-minor-mode and some technical discussion first. But anyway, I
>> will not have time to work on this again before June.
>
> I for one need to have a clearer picture of what such a minor mode
> would really do, without getting prematurily lost in the details of
> possible implementations.

Its just a better and smarter outshine-mode (major-mode agnostic "Org
look&feel" for programming modes). 

> Once we have a clear goal, let's build a small list of steps, then
> submit a patch for the first step.  (We can then work on a separate
> branch for the whole feature, and spend months on this branch.)

The goal for org-minor-mode would be the same like for outshine - give
programming mode buffers the same look&feel like Org-mode buffers. This
works already quite good with outshine, I structure all my elisp,
picolisp, R, CSS (etc etc) files like Org files, and when I switch between
them and Org-files I hardly notice a difference wrt navigation, folding,
structure editing, fontification etc. 

But the backend for outshine is outline-(minor)-mode. Org-mode does some
things better than outline-mode, and tons of things outline-mode can't
do. 

> If the change is relying on too many external libraries, it is not
> very likely to happen.

It should end up as one new library (e.g. omm.el) that is part of
Org-mode, replacing both orgstruct-mode and outshine. 

> The feature seems to bring interest, but let's refrain from spending
> our enthusiasm in early and lengthy discussions that nobody can fully
> follow, especially if you won't have time to hack something before
> June...

outshine, outorg and navi-mode are all in the mid-range of popular melpa
packages, so there seems to be some real demand ...

> So I really don't want to play the cold fish here, but let's move
> with code and small steps rather than discussions and big goals.

I started with code (https://github.com/tj64/omm), but faced the
fundamental problem of hardcoded regexps (^, $, and \\*) all over the
Org sources that make Org functions fail on outcommented headers and in
outcommented text sections in general. 

The goals, ideas and even implementations (outshine, orgstruct) are
already there, a first intent to merge them into one library exists
(omm.el), but what to do about this core problem?

-- 
cheers,
Thorsten

Re: [RFC] Org Minor Mode?

[Bastien]

Hi Thorsten,

Thorsten Jolitz <tjolitz@gmail.com> writes:

>> I for one need to have a clearer picture of what such a minor mode
>> would really do, without getting prematurily lost in the details of
>> possible implementations.
>
> Its just a better and smarter outshine-mode (major-mode agnostic "Org
> look&feel" for programming modes).

But what would it *do*?  Can you give a simple example of a specific
feature?  The one about editing source code in buffers other than Org
buffers, maybe?

> outshine, outorg and navi-mode are all in the mid-range of popular melpa
> packages, so there seems to be some real demand ...

No doubt!

> I started with code (https://github.com/tj64/omm),

(FWIW I don't think omm.el is a really good name, it's hard to guess
what it is supposed to do.  You could use org-minor-mode.el and keep
omm- as a prefix?)

> but faced the fundamental problem of hardcoded regexps (^, $, and
> \\*) all over the Org sources that make Org functions fail on
> outcommented headers and in outcommented text sections in general.
>
> The goals, ideas and even implementations (outshine, orgstruct) are
> already there, a first intent to merge them into one library exists
> (omm.el), but what to do about this core problem?

Circumvent it?  Instead of try to adapt tons of Org features so that
they run into other modes, we could try to emulate them in temporary
buffers, where the peculiarities of the origin mode do not prevent
Org functions from running -- see for example how `org-open-at-point'
deals with links in comments.  This could be generalized to, e.g.,
handle lists in Emacs lisp comments.

-- 
 Bastien

Re: [RFC] Org Minor Mode?

[Thorsten Jolitz]

Bastien <bzg@gnu.org> writes:

Hi Bastien,

> Thorsten Jolitz <tjolitz@gmail.com> writes:
>
>>> I for one need to have a clearer picture of what such a minor mode
>>> would really do, without getting prematurily lost in the details of
>>> possible implementations.
>>
>> Its just a better and smarter outshine-mode (major-mode agnostic "Org
>> look&feel" for programming modes).
>
> But what would it *do*?  Can you give a simple example of a specific
> feature?  The one about editing source code in buffers other than Org
> buffers, maybe?

What outshine does already is the navigation, structure editing,
visibility cycling and fontification stuff in a major-mode agnostic
way. I would like to have the "Org action commands" too, i.e. all those
functions that make Org headers 'smart':

- todos
- priorities
- tags
- timestamps
- clocking
- properties
- ...

There is no real reason why the Org functions could not work on outside
of Org-mode since they a fundamentally based on regexp, only that those
regexps constants are hardcoded (^,$, and \\*) and the functions contain
many hardcoded regesp snippets too - thus they cannot deal with
outcommnented headers in programming buffers.

>> outshine, outorg and navi-mode are all in the mid-range of popular melpa
>> packages, so there seems to be some real demand ...
>
> No doubt!
>
>> I started with code (https://github.com/tj64/omm),
>
> (FWIW I don't think omm.el is a really good name, it's hard to guess
> what it is supposed to do.  You could use org-minor-mode.el and keep
> omm- as a prefix?)

that would be no problem

>> but faced the fundamental problem of hardcoded regexps (^, $, and
>> \\*) all over the Org sources that make Org functions fail on
>> outcommented headers and in outcommented text sections in general.
>>
>> The goals, ideas and even implementations (outshine, orgstruct) are
>> already there, a first intent to merge them into one library exists
>> (omm.el), but what to do about this core problem?
>
> Circumvent it?  Instead of try to adapt tons of Org features so that
> they run into other modes, we could try to emulate them in temporary
> buffers, where the peculiarities of the origin mode do not prevent
> Org functions from running -- see for example how `org-open-at-point'
> deals with links in comments.  This could be generalized to, e.g.,
> handle lists in Emacs lisp comments.

Thats currently possible with outorg.el, M-# M-# on a outshine subtree
or buffer is just the reverse of C-c ' on a source-block - it offers the
subtree of buffer in a temporary *outorg-edit-buffer* in full Org-mode
with the comment-section converted to text and the source-code enclosed
in source-blocks. 

So this works quite well already. But as outshine evolves, me and others
would like to have it more powerful and even more similar to
Org-mode. Then I start reimplementing features based on outline and
extensions, knowing that they are already there in Org-mode,
sophisticated and well tested. Thats a bit frustrating, especially since
it seems that its just a few chars in the regexps that inhibit using
Org-mode functions outside of org-mode. This would be easy to fix in a
small library, but since Org-mode is so big, this is really a political
thing since changing the regexps would cause more than 600 changes in
the sources and then require a (slightly) different approach to writing
regexps in the future. There is definitely a cost, but the gain would be
considerable too ...

-- 
cheers,
Thorsten

Re: [RFC] Org Minor Mode?

[Bastien]

Hi Thorsten

Thorsten Jolitz <tjolitz@gmail.com> writes:

> Thats currently possible with outorg.el, M-# M-# on a outshine subtree
> or buffer is just the reverse of C-c ' on a source-block - it offers the
> subtree of buffer in a temporary *outorg-edit-buffer* in full Org-mode
> with the comment-section converted to text and the source-code enclosed
> in source-blocks.

(Did you look at `org-open-at-point' and the way it handles link in
comments?  I don't think M-# M-# does the same.)

> So this works quite well already. But as outshine evolves, me and others
> would like to have it more powerful and even more similar to
> Org-mode. Then I start reimplementing features based on outline and
> extensions, knowing that they are already there in Org-mode,
> sophisticated and well tested. Thats a bit frustrating, especially since
> it seems that its just a few chars in the regexps that inhibit using
> Org-mode functions outside of org-mode. This would be easy to fix in a
> small library, but since Org-mode is so big, this is really a political
> thing since changing the regexps would cause more than 600 changes in
> the sources and then require a (slightly) different approach to writing
> regexps in the future. There is definitely a cost, but the gain would be
> considerable too ...

This is not about the number of lines to be edited or the man-hour we
need to spend on this.  Rather about performance and maintainability.

-- 
 Bastien

Re: [RFC] Org Minor Mode?

[Thorsten Jolitz]

Bastien <bzg@gnu.org> writes:

Hi Bastien,

> Thorsten Jolitz <tjolitz@gmail.com> writes:
>
>> Thats currently possible with outorg.el, M-# M-# on a outshine subtree
>> or buffer is just the reverse of C-c ' on a source-block - it offers the
>> subtree of buffer in a temporary *outorg-edit-buffer* in full Org-mode
>> with the comment-section converted to text and the source-code enclosed
>> in source-blocks.
>
> (Did you look at `org-open-at-point' and the way it handles link in
> comments?  I don't think M-# M-# does the same.)

I looked at it, but to be honest, could not figure out what you are
referring to. Could you be more specific please (line number or so?)

>> There is definitely a cost, but the gain would be considerable too

> This is not about the number of lines to be edited or the man-hour we
> need to spend on this.  Rather about performance and maintainability.

Ok, I can see that the function call necessary to build the regexps
dynamically can be a problem here. 

Maybe instead of function calls some Backslash Constructs can be used to
derive regexp patterns that replace "^" and "$" in a way that they match
all or at least many comment-syntax (whats the plural of syntax?)?

,--------------------------------------------------------------------
| '\scode'
|     matches any character whose syntax is code. Here code is a
|     character that represents a syntax code: thus, 'w' for word
|     constituent, '-' for whitespace, '(' for open parenthesis, etc.
|     To represent whitespace syntax, use either '-' or a space
|     character. See Syntax Class Table, for a list of syntax codes
|     and the characters that stand for them.
`--------------------------------------------------------------------

e.g.

,---------------------------------------------------------------------
| Comment starters: '<'
| Comment enders: '>'
|     Characters used in various languages to delimit comments. Human
|     text has no comment characters. In Lisp, the semicolon (';')
|     starts a comment and a newline or formfeed ends one.
| [...]
| Generic comment delimiters: '!'
|     Characters that start or end a special kind of comment. Any
|     generic comment delimiter matches any generic comment delimiter,
|     but they cannot match a comment starter or comment ender;
|     generic comment delimiters can only match each other.
|    
|     This syntax class is primarily meant for use with the
|     syntax-table text property (see Syntax Properties). You can mark
|     any range of characters as forming a comment, by giving the
|     first and last characters of the range syntax-table properties
|     identifying them as generic comment delimiters.
`---------------------------------------------------------------------

For the Org star "*" maybe one could define a new category

,--------------------------------------------------------------------
| Categories provide an alternate way of classifying characters
| syntactically. You can define several categories as needed, then
| independently assign each character to one or more categories.
| Unlike syntax classes, categories are not mutually exclusive; it is
| normal for one character to belong to several categories.
`--------------------------------------------------------------------

and match them with

,---------------------------------------------------------------------
| '\cc'
|     matches any character whose category is c. Here c is a character
|     that represents a category: thus, 'c' for Chinese characters or
|     'g' for Greek characters in the standard category table. You can
|     see the list of all the currently defined categories with M-x
|     describe-categories <RET>. You can also define your own
|     categories in addition to the standard ones using the
|     define-category function (see Categories).
`---------------------------------------------------------------------

In the beginning, that category would only consist of (* ;).

This is quite low level and I haven't done anything on this level yet,
but it might be a way to stick with performant constant regexp strings,
but make them more general.

-- 
cheers,
Thorsten

Re: [RFC] Org Minor Mode?

[Bastien]

Hi Thorsten,

Thorsten Jolitz <tjolitz@gmail.com> writes:

> This is quite low level and I haven't done anything on this level yet,
> but it might be a way to stick with performant constant regexp strings,
> but make them more general.

That's an idea -- but the one I wanted to explore is this: instead of
running Org functions in the current buffer (e.g., an emacs-lisp-mode
buffer), the functions would run transparently in a temporary buffer
before updating the current one.

For example, see this code in `org-open-at-point':

;; Exception n°2: links in comments.
((eq type 'comment)
 (let ((string-rear (replace-regexp-in-string
		     "^[ \t]*# [ \t]*" ""
		     (buffer-substring (point) (line-beginning-position))))
       (string-front (buffer-substring (point) (line-end-position))))
   (with-temp-buffer
     (let ((org-inhibit-startup t)) (org-mode))
     (insert value)
     (goto-char (point-min))
     (when (and (search-forward string-rear nil t)
		(search-forward string-front (line-end-position) t))
       (goto-char (match-beginning 0))
       (org-open-at-point)
       (when (string= string-rear "") (forward-char))))))

Obviously, this is a bit fragile and just "good enough" for this
defun, but we could generalize it and make it more robust.

To illustrate this way of approaching the problem of "using Org in
non-Org buffers", consider `org-move-item-up': in an elisp comment,
we would copy the comment paragraph at point in `with-temp-buffer',
uncomment the buffer, call `org-move-item-up', then comment out the
content and send it back to the original buffer as a replacement.

I've not studied this idea thoroughly, and this may feels clumsy
first, but if it spares us with the need to generalize org-mode to
the point that org-mode is not really a separate mode anymore, it
might be worst digging into this direction.

Hope this is clearer now!

-- 
 Bastien

Re: [RFC] Org Minor Mode?

[Thorsten Jolitz]

Bastien <bzg@gnu.org> writes:

Hi Bastien,

> Thorsten Jolitz <tjolitz@gmail.com> writes:
>
>> This is quite low level and I haven't done anything on this level yet,
>> but it might be a way to stick with performant constant regexp strings,
>> but make them more general.
>
> That's an idea -- but the one I wanted to explore is this: instead of
> running Org functions in the current buffer (e.g., an emacs-lisp-mode
> buffer), the functions would run transparently in a temporary buffer
> before updating the current one.
>
> For example, see this code in `org-open-at-point':
>
> ;; Exception n°2: links in comments.
> ((eq type 'comment)
>  (let ((string-rear (replace-regexp-in-string
> 		     "^[ \t]*# [ \t]*" ""
> 		     (buffer-substring (point) (line-beginning-position))))
>        (string-front (buffer-substring (point) (line-end-position))))
>    (with-temp-buffer
>      (let ((org-inhibit-startup t)) (org-mode))
>      (insert value)
>      (goto-char (point-min))
>      (when (and (search-forward string-rear nil t)
> 		(search-forward string-front (line-end-position) t))
>        (goto-char (match-beginning 0))
>        (org-open-at-point)
>        (when (string= string-rear "") (forward-char))))))
>
> Obviously, this is a bit fragile and just "good enough" for this
> defun, but we could generalize it and make it more robust.

oh, I see, this must be a very recent addition, did not have this
section in my version (-> must update). 

There was a recent discussion on emacs-devel about adding a generic
`inverse-comment' function to core emacs, this would be a nice use case
(I would need that badly for outorg too, would enable it to deal with
special multi-line comments like those in C or Java). Unfortunately they
got lost in discussing how to deal with 'inline' comments, e.g. those
after a line.

If the devel's could be convinced to drop these special-case
requirements and add a robust generic core function to emacs that
inverts outcommented lines (single and multi), it would be very easy to
implement this idea in a major-mode agnostic way. Would be even better
if the function returned start and end point of the inverted region so
one could act on it somehow afterwards. 

There have been proposols and even implementations, but no conclusion
yet, maybe its worth to enter that discussion.  

This could be a solution - let many of the high-level commands check if
- org-minor-mode is active 
- they are in a comment
and if so use a temp buffer to act on the uncommented text. 

One problem is that more and more Org functions seems to get redefined
in terms of the new parser functionality, and obviously then can't be
used anymore outside org-mode. 

I realized this when looking at `org-open-at-point', e.g. this snippet:

#+begin_src emacs-lisp
  (...)    
  (let* ((context (org-element-context)) type)
    ;; On an unsupported type, check if point is contained within
    ;; a support one.
    (while (and (not (memq (setq type (org-element-type context))
                           '(headline inlinetask link footnote-definition
                                      footnote-reference timestamp)))
                (setq context (org-element-property :parent context))))
  (...)
#+end_src


Maybe this renders the whole idea of using Org functions outside of
org-mode as useless? For this snippet to work, I assume the whole
buffer must have been parsed successfully and thus must be in valid
Org syntax?

> To illustrate this way of approaching the problem of "using Org in
> non-Org buffers", consider `org-move-item-up': in an elisp comment,
> we would copy the comment paragraph at point in `with-temp-buffer',
> uncomment the buffer, call `org-move-item-up', then comment out the
> content and send it back to the original buffer as a replacement.
>
> I've not studied this idea thoroughly, and this may feels clumsy
> first, but if it spares us with the need to generalize org-mode to
> the point that org-mode is not really a separate mode anymore, it
> might be worst digging into this direction.
>
> Hope this is clearer now!

Yes, thanks! I think I get your idea of rather do some more
conditional action in some high-level user commands and keep the
frequently called low-level functions as performant as possible. 

-- 
cheers,
Thorsten

Re: [RFC] Org Minor Mode?

[Bastien]

Hi Thorsten,

Thorsten Jolitz <tjolitz@gmail.com> writes:

> One problem is that more and more Org functions seems to get redefined
> in terms of the new parser functionality, and obviously then can't be
> used anymore outside org-mode.

That's not a problem if we follow the path I suggest: since the
tempoary buffer we operate in will be in org-mode at some point, the
parser will work there.

-- 
 Bastien

Re: [RFC] Org Minor Mode?

[Thorsten Jolitz]

Bastien <bzg@gnu.org> writes:

Hi Bastien,

> Thorsten Jolitz <tjolitz@gmail.com> writes:
>
>> One problem is that more and more Org functions seems to get redefined
>> in terms of the new parser functionality, and obviously then can't be
>> used anymore outside org-mode.
>
> That's not a problem if we follow the path I suggest: since the
> tempoary buffer we operate in will be in org-mode at some point, the
> parser will work there.

I'm still not sure if this path isn't just the outorg path: copy a
subtree or a whole buffer into a temp-buffer, uncomment the comment
sections, enclose the source-code in source-blocks, and put the buffer
in Org-mode - only that the tmp buffer in this case is not for user
editing but for program execution. 

What will be copied to the temp-buffer? Only the comment-section at
point? The subtree at point? The (outcommented) element at point? Won't
some Org functions fail without the subtree/buffer context? What if text
is inserted as side-effect? E.g state change logs when going from TODO
to done, or even footnotes?

-- 
cheers,
Thorsten

Re: [RFC] Org Minor Mode?

[Bastien]

Hi Thorsten,

Thorsten Jolitz <tjolitz@gmail.com> writes:

> I'm still not sure if this path isn't just the outorg path: copy a
> subtree or a whole buffer into a temp-buffer, uncomment the comment
> sections, enclose the source-code in source-blocks, and put the buffer
> in Org-mode - only that the tmp buffer in this case is not for user
> editing but for program execution.

Yes -- that's the outorg path but transparent to the user, with no
editing in the middle.

> What will be copied to the temp-buffer? Only the comment-section at
> point? The subtree at point? The (outcommented) element at point? Won't
> some Org functions fail without the subtree/buffer context? What if text
> is inserted as side-effect? E.g state change logs when going from TODO
> to done, or even footnotes?

Well, I don't know.  Again, all this feels certainly dirty, but I'm
trying to find something that will feel less awkward than making tons
of Org regexps relative to their contexts, including the context for
non-org-mode buffers...

-- 
 Bastien

Re: [RFC] Org Minor Mode?

[Thorsten Jolitz]

Bastien <bzg@gnu.org> writes:

Hi Bastien,

> Thorsten Jolitz <tjolitz@gmail.com> writes:

>> What will be copied to the temp-buffer? Only the comment-section at
>> point? The subtree at point? The (outcommented) element at point? Won't
>> some Org functions fail without the subtree/buffer context? What if text
>> is inserted as side-effect? E.g state change logs when going from TODO
>> to done, or even footnotes?
>
> Well, I don't know.  Again, all this feels certainly dirty, but I'm
> trying to find something that will feel less awkward than making tons
> of Org regexps relative to their contexts, including the context for
> non-org-mode buffers...

I thought about a very low level solution, i.e. at the syntax table
level, giving "^ ... $" a different meaning, but thats a bit over my
head and I have my doubt if its possible at all. 

BTW, I think I already implemented your idea for when porting the speed
commands to outshine:

,--------------------------------------------------
| (defconst outshine-speed-commands-default
|   '(
|     ("Outline Navigation")
|     ("n" . (outshine-speed-move-safe
|             'outline-next-visible-heading))
|     [...]
|     ("j" . (outshine-use-outorg 'org-goto))
|     ("g" . (outshine-use-outorg 'org-refile))
|     ("Outline Visibility")
|     ("c" . outline-cycle)
|     ("C" . outshine-cycle-buffer)
|     ;; FIXME needs to be improved!
|     (" " . (outshine-use-outorg
|             (lambda ()
|               (message
|                "%s" (substring-no-properties
|                      (org-display-outline-path)))
|                (sit-for 1))
|             'WHOLE-BUFFER-P))
|    [...]
`--------------------------------------------------

This `outshine-use-outorg' function does more or less what you talk
about:

#+begin_src emacs-lisp
(eval-after-load 'outorg
  '(defun outshine-use-outorg (fun &optional whole-buffer-p &rest funargs)
     "Use outorg to call FUN with FUNARGS on subtree.

FUN should be an Org-mode function that acts on the subtree at
point. Optionally, with WHOLE-BUFFER-P non-nil,
`outorg-edit-as-org' can be called on the whole buffer.

Sets the variable `outshine-use-outorg-last-headline-marker' so
that it always contains a point-marker to the last headline this
function was called upon.

The old marker is removed first. Then a new point-marker is
created before `outorg-edit-as-org' is called on the headline."
     (save-excursion
       (unless (outline-on-heading-p)
	 (outline-previous-heading))
       (outshine--set-outorg-last-headline-marker)
       (if whole-buffer-p
	   (outorg-edit-as-org '(4))
	 (outorg-edit-as-org))
       (if funargs
	   (funcall fun funargs)
	 (funcall fun))
       (outorg-copy-edits-and-exit))))
#+end_src

So maybe I should stop insisting on an org-minor-mode, because outshine
and outorg together already do the trick? 

I just thought it would be better, faster and more powerfull if Org's
regexps would be more abstract and Org functions could act directly in
the programming mode buffers. 

-- 
cheers,
Thorsten

Re: [RFC] Org Minor Mode?

[Bastien]

Hi Thorsten,

Thorsten Jolitz <tjolitz@gmail.com> writes:

> So maybe I should stop insisting on an org-minor-mode, because outshine
> and outorg together already do the trick?

Indeed!  (Do you have a screencast demonstrating this?  It's all a bit
abstract when put in words.)

> I just thought it would be better, faster and more powerfull if Org's
> regexps would be more abstract and Org functions could act directly in
> the programming mode buffers.

I'm surely getting old, but "better is the ennemy of good".

If we already have something that fills the needs for org-minor-mode,
then let's advertize this more and spare the work involved by making
things faster... (do you really need to make outshine+outorg faster?)

-- 
 Bastien

Re: [RFC] Org Minor Mode?

[Thorsten Jolitz]

Bastien <bzg@gnu.org> writes:

Hi Bastien,

> Thorsten Jolitz <tjolitz@gmail.com> writes:
>
>> So maybe I should stop insisting on an org-minor-mode, because outshine
>> and outorg together already do the trick?
>
> Indeed!  (Do you have a screencast demonstrating this?  It's all a bit
> abstract when put in words.)

Not yet, but outshine users know that they can use many of the same
speed commands on outshine headers like on org headers. I'll see if I
can upload something to youtube.

>> I just thought it would be better, faster and more powerfull if Org's
>> regexps would be more abstract and Org functions could act directly in
>> the programming mode buffers.
>
> I'm surely getting old, but "better is the ennemy of good".
>
> If we already have something that fills the needs for org-minor-mode,
> then let's advertize this more and spare the work involved by making
> things faster... (do you really need to make outshine+outorg faster?)

My motivation for an org-minor-mode came from situations where I wanted
some feature for outshine, like e.g. the :archive: tag (that keeps
subtrees permanently folded and would be very useful for hiding
commentary subtrees in programming mode files) or visibility cycling for
other elements than headlines, and I figured that it was all there in
Org-mode, but I would have to reimplement it on top of the outline.el
backend. 

After this discussion it seems that reimplementing things is still the
better (and maybe the only viable) option. So lets close this thread,
and I focus on improving outshine/outorg so that they become (not by
name, but by functiionality) a kind of Org minor-mode.

Thanks for your time and input! 

-- 
cheers,
Thorsten