Re Org 9.4 is out. Can you help? // breaking apart Org Mode

Re Org 9.4 is out. Can you help? // breaking apart Org Mode

[William Rankin via Emacs development discussions.]

Hello,

At the request of Bastien I'm sending on these ideas regarding the 
future of Org Mode development. I'm also copying emacs-devel since they 
might be interested too.

Org Mode and Emacs would benefit greatly from the codebase being broken 
apart, not unlike how an antitrust suit breaks apart a big company for 
the good of society!

It is my view that many parts of Org code could be implemented as minor 
modes or independent libraries. This would encourage cleaner, more 
modular and more easy to understand code. It would provide an 
exponential benefit for other elisp programs. And by splitting up the 
codebase you allow contributors more a sense of ownership and emotional 
investment in the things to which they provide their time/effort.

A few suggestions...

* outline

Org Mode builds on top of outline, but those improvements are isolated 
to Org, e.g. Org has wonderful outline cycling, but if someone wants 
outline cycling in another major mode they need to implement this again 
(likely just duplicating Org's existing code). Ideally all of this could 
be ported back to outline itself. This would slim down the Org codebase 
while benefiting all other outline-based major modes.

* orgtbl-mode

This is a good attempt at implementing some of Org's functionality as a 
minor mode. Ideally orgtbl could be ported back to table and enough 
flexibility added to make it compatible with Markdown Mode tables 
(currently implemented with its own table stuff).

* source blocks

Org's source block functionality could be spun off into its own library. 
In theory it could work just like outline (where a major mode defines 
its own heading regexp). A major mode would define its own source block 
delimiter regexpes.

Ideally any major mode writing for a plain text markup format would 
just:
  (require 'source-blocks)
then have all the same functionality of Org source blocks. Any 
improvements would then benefit everyone.

* org-toggle-time-stamp-overlays / org-toggle-link-display

This functionality, although small within Org, could be very nice as 
their own minor modes. Displaying dates/times with custom format is easy 
enough... URLs a bit harder.

I went so far as to try this with varying degrees of success:
https://github.com/rnkn/prettify-date-time-mode
https://github.com/rnkn/prettify-url-mode

* org-link

I see a lot of interest for that Zettelkasten method, with many 
different implementations. What's stopping Org's cross-linking being 
implemented as its own global minor mode, independent of .org files?

* electric-pair-mode

Org currently uses org-emphasize for its emphasis pairs, but could it 
just use electric-pair-mode? Would this prompt some improvements to 
electric-pair-mode? This would benefit everyone.


I don't mean to suggest that the above ideas are things I'm particularly 
hanging out for, I'm just trying to sketch an ideas of beneficial ways 
Org could be broken apart.

Finally, I'm pretty sure breaking apart Org will mean it will be much 
easier to maintain -- it will be far easier to find one or two people 
passionate about maintaining perhaps a source-blocks library than the 
entirety of Org. If Org's development takes this more modular direction, 
where libraries are designed to work independently of the rest of the 
code, then it won't need an elite few people who understand the whole 
codebase.

I hope some of these ideas were either valuable or provide valuable 
discussion.

-- 
William Rankin
https://bydasein.com

~ The single best thing you can do for the world is to delete your 
social media accounts.

Re Org 9.4 is out. Can you help? // breaking apart Org Mode

[William Rankin via General discussions about Org-mode.]

Hello,

At the request of Bastien I'm sending on these ideas regarding the 
future of Org Mode development. I'm also copying emacs-devel since they 
might be interested too.

Org Mode and Emacs would benefit greatly from the codebase being broken 
apart, not unlike how an antitrust suit breaks apart a big company for 
the good of society!

It is my view that many parts of Org code could be implemented as minor 
modes or independent libraries. This would encourage cleaner, more 
modular and more easy to understand code. It would provide an 
exponential benefit for other elisp programs. And by splitting up the 
codebase you allow contributors more a sense of ownership and emotional 
investment in the things to which they provide their time/effort.

A few suggestions...

* outline

Org Mode builds on top of outline, but those improvements are isolated 
to Org, e.g. Org has wonderful outline cycling, but if someone wants 
outline cycling in another major mode they need to implement this again 
(likely just duplicating Org's existing code). Ideally all of this could 
be ported back to outline itself. This would slim down the Org codebase 
while benefiting all other outline-based major modes.

* orgtbl-mode

This is a good attempt at implementing some of Org's functionality as a 
minor mode. Ideally orgtbl could be ported back to table and enough 
flexibility added to make it compatible with Markdown Mode tables 
(currently implemented with its own table stuff).

* source blocks

Org's source block functionality could be spun off into its own library. 
In theory it could work just like outline (where a major mode defines 
its own heading regexp). A major mode would define its own source block 
delimiter regexpes.

Ideally any major mode writing for a plain text markup format would 
just:
  (require 'source-blocks)
then have all the same functionality of Org source blocks. Any 
improvements would then benefit everyone.

* org-toggle-time-stamp-overlays / org-toggle-link-display

This functionality, although small within Org, could be very nice as 
their own minor modes. Displaying dates/times with custom format is easy 
enough... URLs a bit harder.

I went so far as to try this with varying degrees of success:
https://github.com/rnkn/prettify-date-time-mode
https://github.com/rnkn/prettify-url-mode

* org-link

I see a lot of interest for that Zettelkasten method, with many 
different implementations. What's stopping Org's cross-linking being 
implemented as its own global minor mode, independent of .org files?

* electric-pair-mode

Org currently uses org-emphasize for its emphasis pairs, but could it 
just use electric-pair-mode? Would this prompt some improvements to 
electric-pair-mode? This would benefit everyone.


I don't mean to suggest that the above ideas are things I'm particularly 
hanging out for, I'm just trying to sketch an ideas of beneficial ways 
Org could be broken apart.

Finally, I'm pretty sure breaking apart Org will mean it will be much 
easier to maintain -- it will be far easier to find one or two people 
passionate about maintaining perhaps a source-blocks library than the 
entirety of Org. If Org's development takes this more modular direction, 
where libraries are designed to work independently of the rest of the 
code, then it won't need an elite few people who understand the whole 
codebase.

I hope some of these ideas were either valuable or provide valuable 
discussion.

-- 
William Rankin
https://bydasein.com

~ The single best thing you can do for the world is to delete your 
social media accounts.

Re: Re Org 9.4 is out. Can you help? // breaking apart Org Mode

[Nicholas Savage]

Hi there,

I'm new to the development scene (contributed my first patch to Emacs last week or so) and a new developer in general who has been looking to get involved. I saw Bastien's post on Hacker News yesterday and thought that if Org Mode is looking for contributors it would be a good place for me, since I use Org Mode frequently. I have a rudimentary understanding of elisp, but I'm looking to improve by actually contributing.

I'm just wondering about some of these ideas, mostly from an administrative point of view. I think they're fantastic ideas, such as moving code from Org Mode back up to Outline mode. I'm just looking to wrap my head around how this is supposed to work (I already understand submitting patches and can read the READMEs, so don't worry about that).

If I am looking to do that though, would I be submitting a patch both to Emacs and one to Org Mode? At what point are Org Mode commits merged into Emacs? I guess my concern is that these changes could be breaking for users of Org Mode if they're not also using the most recent master of Emacs as well.

Nick

On Tue, Sep 15, 2020, at 06:05, William Rankin via General discussions about Org-mode. wrote:
> Hello,
> 
> At the request of Bastien I'm sending on these ideas regarding the 
> future of Org Mode development. I'm also copying emacs-devel since they 
> might be interested too.
> 
> Org Mode and Emacs would benefit greatly from the codebase being broken 
> apart, not unlike how an antitrust suit breaks apart a big company for 
> the good of society!
> 
> It is my view that many parts of Org code could be implemented as minor 
> modes or independent libraries. This would encourage cleaner, more 
> modular and more easy to understand code. It would provide an 
> exponential benefit for other elisp programs. And by splitting up the 
> codebase you allow contributors more a sense of ownership and emotional 
> investment in the things to which they provide their time/effort.
> 
> A few suggestions...
> 
> * outline
> 
> Org Mode builds on top of outline, but those improvements are isolated 
> to Org, e.g. Org has wonderful outline cycling, but if someone wants 
> outline cycling in another major mode they need to implement this again 
> (likely just duplicating Org's existing code). Ideally all of this could 
> be ported back to outline itself. This would slim down the Org codebase 
> while benefiting all other outline-based major modes.
> 
> * orgtbl-mode
> 
> This is a good attempt at implementing some of Org's functionality as a 
> minor mode. Ideally orgtbl could be ported back to table and enough 
> flexibility added to make it compatible with Markdown Mode tables 
> (currently implemented with its own table stuff).
> 
> * source blocks
> 
> Org's source block functionality could be spun off into its own library. 
> In theory it could work just like outline (where a major mode defines 
> its own heading regexp). A major mode would define its own source block 
> delimiter regexpes.
> 
> Ideally any major mode writing for a plain text markup format would 
> just:
>   (require 'source-blocks)
> then have all the same functionality of Org source blocks. Any 
> improvements would then benefit everyone.
> 
> * org-toggle-time-stamp-overlays / org-toggle-link-display
> 
> This functionality, although small within Org, could be very nice as 
> their own minor modes. Displaying dates/times with custom format is easy 
> enough... URLs a bit harder.
> 
> I went so far as to try this with varying degrees of success:
> https://github.com/rnkn/prettify-date-time-mode
> https://github.com/rnkn/prettify-url-mode
> 
> * org-link
> 
> I see a lot of interest for that Zettelkasten method, with many 
> different implementations. What's stopping Org's cross-linking being 
> implemented as its own global minor mode, independent of .org files?
> 
> * electric-pair-mode
> 
> Org currently uses org-emphasize for its emphasis pairs, but could it 
> just use electric-pair-mode? Would this prompt some improvements to 
> electric-pair-mode? This would benefit everyone.
> 
> 
> I don't mean to suggest that the above ideas are things I'm particularly 
> hanging out for, I'm just trying to sketch an ideas of beneficial ways 
> Org could be broken apart.
> 
> Finally, I'm pretty sure breaking apart Org will mean it will be much 
> easier to maintain -- it will be far easier to find one or two people 
> passionate about maintaining perhaps a source-blocks library than the 
> entirety of Org. If Org's development takes this more modular direction, 
> where libraries are designed to work independently of the rest of the 
> code, then it won't need an elite few people who understand the whole 
> codebase.
> 
> I hope some of these ideas were either valuable or provide valuable 
> discussion.
> 
> -- 
> William Rankin
> https://bydasein.com
> 
> ~ The single best thing you can do for the world is to delete your 
> social media accounts.
> 
>

Re: Re Org 9.4 is out. Can you help? // breaking apart Org Mode

[Russell Adams]

On Tue, Sep 15, 2020 at 09:00:39AM -0400, Nicholas Savage wrote:
> I'm new to the development scene (contributed my first patch to
> Emacs last week or so) and a new developer in general who has been
> looking to get involved. I saw Bastien's post on Hacker News
> yesterday and thought that if Org Mode is looking for contributors
> it would be a good place for me, since I use Org Mode frequently. I
> have a rudimentary understanding of elisp, but I'm looking to
> improve by actually contributing.

Nicholas,

I'm in much the same position. I've offered to help and trying to
participate. My elisp is ok, but I have little practical experience.

From the code standpoint my concern for several years has been that
Org is so powerful, too powerful, that every edge case asks attention
to be added into core. There are so many things Org has been expanded
to do that were outside of the original scope. That growth can't be
sustained with volunteer time forever.

Rather than a huge refactoring or pushing code back into other Emacs
modes, my thought was that Org should be trimmed into the "core" of
Org functions and that other things should be implemented as modules
available in MELPA outside of the official Org core. That way the
limited maintainer time can be focused on the core of Org and
maintaining strong interfaces for components made and maintained by
others.

I understand that there may be some effort already underway to push
items into external packages.

Perhaps this is a better place to start than a huge refactoring
effort? Also note that I feel like the constant deluge of additions
prohibits refactoring and code improvement in the core. That's my
outside perspective though.


------------------------------------------------------------------
Russell Adams                            RLAdams@AdamsInfoServ.com

PGP Key ID:     0x1160DCB3           http://www.adamsinfoserv.com/

Fingerprint:    1723 D8CA 4280 1EC9 557F  66E8 1154 E018 1160 DCB3

Re: Re Org 9.4 is out. Can you help? // breaking apart Org Mode

[Bastien]

Hi Russell,

Russell Adams <RLAdams@AdamsInfoServ.Com> writes:

> Rather than a huge refactoring or pushing code back into other Emacs
> modes, my thought was that Org should be trimmed into the "core" of
> Org functions and that other things should be implemented as modules
> available in MELPA outside of the official Org core. That way the
> limited maintainer time can be focused on the core of Org and
> maintaining strong interfaces for components made and maintained by
> others.

Yes.  

I agree with this goal, as long as Org modules are maintained too.

Picking up the example I gave above in this thread of modularizing
org-table.el, it will be a good think iff it gets as much attention
than Org's core itself.

2 cents,

-- 
 Bastien

Re: Re Org 9.4 is out. Can you help? // breaking apart Org Mode

[Dr. Arne Babenhauserheide]

[-- Attachment #1: Type: text/plain, Size: 1804 bytes --]


Bastien <bzg@gnu.org> writes:

>> Rather than a huge refactoring or pushing code back into other Emacs
>> modes, my thought was that Org should be trimmed into the "core" of
>> Org functions and that other things should be implemented as modules
>> available in MELPA outside of the official Org core. That way the
>> limited maintainer time can be focused on the core of Org and
>> maintaining strong interfaces for components made and maintained by
>> others.

> I agree with this goal, as long as Org modules are maintained too.
>
> Picking up the example I gave above in this thread of modularizing
> org-table.el, it will be a good think iff it gets as much attention
> than Org's core itself.

Keep in mind that this will make it much harder to change org-api.

I have some experience with Mecurial extensions, as a maintainer of
those, and basically every extension which is not shipped with Mercurial
(those live in the same source tree) is broken from time to time.

I had to give up one extension and two are currently broken (but not yet
really given up), because I could not keep up with the changes in
Mercurial, simply due to reduced free time.

So while I think that having a stronger separation, I only see a good
case for moving parts out of the org-mode source tree when there is
considerably more activity in those parts than in the core. That can
allow faster release-cycles than org itself, since fewer parts have to
be tested.

Stuff that’s moved out while it does not have its own community is in
danger of becoming dead code that must not be broken, but is much less
convenient to test (and the tests harder to automate) when stuff in org
core changes.

Best wishes,
Arne
-- 
Unpolitisch sein
heißt politisch sein
ohne es zu merken

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 1125 bytes --]

Re: Re Org 9.4 is out. Can you help? // breaking apart Org Mode

[Bastien]

Hi Nicholas,

"Nicholas Savage" <nick@nicksavage.ca> writes:

> If I am looking to do that though, would I be submitting a patch both
> to Emacs and one to Org Mode?

If the patch is against Org, submit it first to emacs-orgmode@gnu.org.

If the patch is integrated into Org it will be merged into Emacs core
at some point.  If the change in the patch, allows for an improvement
in Emacs, then send a patch to the emacs-devel list.

E.g.: if you manage to make to modularize org-table.el so that it can
work on both .org and .md files, first propose the change to this list
then to the markdown-mode developers on Emacs.

Thanks!

-- 
 Bastien

Re: Re Org 9.4 is out. Can you help? // breaking apart Org Mode

[Thorsten Jolitz]

William Rankin via "Emacs development discussions."
<emacs-devel@gnu.org> writes:

Hello,

> At the request of Bastien I'm sending on these ideas regarding the
> future of Org Mode development. I'm also copying emacs-devel since
> they might be interested too.
>
> Org Mode and Emacs would benefit greatly from the codebase being
> broken apart, not unlike how an antitrust suit breaks apart a big
> company for the good of society!

[...]

I do have some hands-on experience with this, although quite some time
ago. When writing the "outshine-suite" (outshine.el, outorg.el
navi-mode.el), it was all about bringing org-mode functionality to
programming modes, or any other major mode that is not org-mode, but
with the org-mode magic acting on outshine headings (outcommented org
headings).

I had to port a lot of org-mode functionality to outshine then (based on
existing outline extensions like outline-magic), and had quite a few
user requests like "I want org-mode feature xyz in outshine too".

I quickly realized that this was an uphill battle and came up with a
'outshine-use-outorg' function, that simply first converts the outshine
buffer to org-mode, executes the requested org command, then converts
the buffer back to outshine. I created a skeleton file with skeletons
for all org-mode commands at that time, and implemented a few, leaving
the others to any contributors that feel a need for the feature. 

Here is what I remember about that process of porting org-mode
functionality to outshine:

1. org-mode is clearly not written in "library-style". 
2. many org commands don't expose their parameters in a function
signature, but rather ask the user for them with some interactive prompt
3. some org commands are "stateful", like clocking e.g., and it might be
hard to preserve the state without having the related org buffer around 
4. many org commands don't do just one thing, they are smart in a way
that makes the user happy, but the programmer who wants to reuse them
unhappy. 

I remember that some functionality was straight forward to port (like
org speed commands), and other not so much (although org-mode had some
enhanced outline functions, there were reasons that I used and improved
the outline-magic versions iirc).

This project could be quite a lot of work (adressing points 2-4 above)
imo, but sounds very nice. 
 
Just my 2 cents ...

-- 
cheers,
Thorsten

Re: Re Org 9.4 is out. Can you help? // breaking apart Org Mode

[Bastien]

Hi William,

thanks a lot for bringing this up.

Of course, Org would benefit from code cleanup and code refactoring.

And yes, we can collectively push toward (1) modularizing Org a little
more, (2) making Org features better interact with Emacs core features
and (3) integrating some of Org's features into Emacs core as Emacs
features.

IMHO the good examples you give fall into one of the category above,
and I think such efforts are likely to happen in that order: 1, 2, 3.

The better way to make this happen is to start a discussion with a
patch explaining how it makes 1, 2 or 3, then discussing the patch
here on this list - the smaller the better.

If you cannot make a patch, first discuss your idea, and once the
implementation seems clear, call for help by using a mail header:

  X-Woof-Help: Help with making X a new module

Thanks,

-- 
 Bastien

RE: Re Org 9.4 is out. Can you help? // breaking apart Org Mode

[arthur miller]

[-- Attachment #1: Type: text/plain, Size: 1658 bytes --]

Not long time ago I posted a bug report about superscripts and subscripts not rendered when in-between italics markings, '/'. I would definitely like to see that code, and rest for  prettie-fying entities factored out into a minor mode that can be activated in any Emacs  buffer.

What do you think, is it to much work and where can you point out (just generally) where to look in the source for the code responsible for that?



-------- Originalmeddelande --------
Från: Bastien <bzg@gnu.org>
Datum: 2020-09-23 10:21 (GMT+01:00)
Till: "William Rankin via General discussions about Org-mode." <emacs-orgmode@gnu.org>
Kopia: William Rankin <william@bydasein.com>, emacs-devel@gnu.org
Ämne: Re: Re Org 9.4 is out. Can you help? // breaking apart Org Mode

Hi William,

thanks a lot for bringing this up.

Of course, Org would benefit from code cleanup and code refactoring.

And yes, we can collectively push toward (1) modularizing Org a little
more, (2) making Org features better interact with Emacs core features
and (3) integrating some of Org's features into Emacs core as Emacs
features.

IMHO the good examples you give fall into one of the category above,
and I think such efforts are likely to happen in that order: 1, 2, 3.

The better way to make this happen is to start a discussion with a
patch explaining how it makes 1, 2 or 3, then discussing the patch
here on this list - the smaller the better.

If you cannot make a patch, first discuss your idea, and once the
implementation seems clear, call for help by using a mail header:

  X-Woof-Help: Help with making X a new module

Thanks,

--
 Bastien


[-- Attachment #2: Type: text/html, Size: 2494 bytes --]

RE: Re Org 9.4 is out. Can you help? // breaking apart Org Mode

[Ihor Radchenko]

> What do you think, is it to much work and where can you point out (just generally) where to look in the source for the code responsible for that?

Sub/superscripts are all dumped inside org.el (together with most of
font-lock-related code).

arthur miller <arthur.miller@live.com> writes:

> Not long time ago I posted a bug report about superscripts and subscripts not rendered when in-between italics markings, '/'. I would definitely like to see that code, and rest for  prettie-fying entities factored out into a minor mode that can be activated in any Emacs  buffer.
>
> What do you think, is it to much work and where can you point out (just generally) where to look in the source for the code responsible for that?
>
>
>
> -------- Originalmeddelande --------
> Från: Bastien <bzg@gnu.org>
> Datum: 2020-09-23 10:21 (GMT+01:00)
> Till: "William Rankin via General discussions about Org-mode." <emacs-orgmode@gnu.org>
> Kopia: William Rankin <william@bydasein.com>, emacs-devel@gnu.org
> Ämne: Re: Re Org 9.4 is out. Can you help? // breaking apart Org Mode
>
> Hi William,
>
> thanks a lot for bringing this up.
>
> Of course, Org would benefit from code cleanup and code refactoring.
>
> And yes, we can collectively push toward (1) modularizing Org a little
> more, (2) making Org features better interact with Emacs core features
> and (3) integrating some of Org's features into Emacs core as Emacs
> features.
>
> IMHO the good examples you give fall into one of the category above,
> and I think such efforts are likely to happen in that order: 1, 2, 3.
>
> The better way to make this happen is to start a discussion with a
> patch explaining how it makes 1, 2 or 3, then discussing the patch
> here on this list - the smaller the better.
>
> If you cannot make a patch, first discuss your idea, and once the
> implementation seems clear, call for help by using a mail header:
>
>   X-Woof-Help: Help with making X a new module
>
> Thanks,
>
> --
>  Bastien