Is Org really so simple?

[Jean Louis <bugs@gnu.support>]

* Ihor Radchenko <yantar92@gmail.com> [2020-11-23 17:18]:
  :PROPERTIES:
  :CREATED:  [2020-11-23 Mon 18:42]
  :ID:       edebb3e7-e755-4ecc-a5e8-a3353a3f5fd0
  :END:
> Dear Jean Louis,
> 
> Your description of the database reminds me how org-roam handles the
> files - it also uses an external database for linking and allows quick
> incremental search that does not really depend on where the
> files/headings are stored.

Sounds good, I can see there is graph database used.

> However, what you are talking about is against org-mode philosophy,
> as I know it.

Only philosophy I know is that it is plain text. Is there any official
philosophy? I have no idea, at least manual does not give me
references. I cannot find "philosophy", send me references.

It says "to keep simple things simple". But Org is far far from being
simple any more. It offers good principles, paradigms and people built
many enhancements upon those. Speedily it becomes way much more than
simple.

Headings do not look any more as headings, it looks like pieces of
code to a person that is new. Properties, tags, clocks, schedule,
deadline, all that tries to store itself under specific heading. There
is easily too much of it and things are not simple any more.

> Currently, the dev's stance is that org files must be
> self-sufficient.

There is no compact principle there practically. Anything is
possible. That Org files are not practically self-sufficient shows the
fact that there are 129 Emacs packages in one Org distribution. 

> Org-mode should not depend on external database to manage the org
> files and operations with them. Everything must be plain text!

Question is what is meant by database, it can be anything. One can
save LISP data. Recent files, desktop, eww bookmarks, init.el or
.emacs file are also all similar databases, there is the underused
EIEIO with persistent stuff that represent built-in database
functionality.

That Org files are not self-sufficient shows the demand that there is
almost no Org user who does not have add-ons, packages, modifications,
configurations.

Would it be really self-sufficient there would be no development going
on, right?

Babel executions clearly show that Org is not self sufficient and
depends on number of external software.

I don't mind of philosophy, in fact I would like that philosophy is
really that what it wanted to be, but that time is over.

I am just pointing out that it is by many means not self sufficient.

Is by default LaTeX export enabled? I think it is. How big is the
LaTeX package? It is huge, and Org depends on it for export.

Thus Org is far far from being self-sufficient.

Almost every system has GDBM database, if Emacs would have bindings to
GDBM, there would be so much less of development in general for many
various packages and Emacs would be expanding faster and easier.

In fact I think that author and initial developers could not predict
at the time where the Org goes and that speaking of self-sufficient
and "plain text" only is history.

Before I found out about Org I was using back in time `hnb' in console
to track various planning tasks. It works nice and simple. That is
really self sufficient. Org definitely not.

HNB - Hierarchical Notebook
http://hnb.sourceforge.net/

In the mean time I have created database where I can store TODO,
Notes, Calls, SMS, People, Invoices, Groups, Mailing Lists and so on,
and made my own shell and Perl interfaces to it. And I used to manage
it through: GeDaFe - PostgreSQL Generic Database Interface
http://gedafe.github.io/doc/gedafe-sql.en.html and this was and is
hierarchical or better graph knowledge management and relational
database.

Creating simple table in the database automatically helped me to
manage that table. It is trivial to create NOTES table with schedule
date, clock in, TODO or other conditions and tags. The interface is
just there and automatic to whatever table I create the interface is
there to add/modify/delete/search/refine records. That is what I would
say "simple" and keeping things simple and indefinitely extensible
without modification of software. The fundamental GeDaFe principle I
would like to try to implement in Emacs. And same database I use for
web interface I am using also within Emacs.

GeDaFe principle is following:

- define your data (like handling notes, TODO, or executing scripts within notes)

- work and forget about any underlying functions,
  add/create/delete/modify/index or search, make reports with simple exports

- expand with more definitions of data when necessary (like add
  various properties, or other data tables, like contacts, invoices,
  etc.) and repeat the process.

Org also shows that it does not hold "Notes" only, it holds more than
that, I have written average book size technical documents with
it. Only just one part of the document is printed from one single node
that belongs to single project among many. People use such documents
on the ground. My use case is not for simple notes.

A node in a tree can be anything, and Org enhancements develop by that
principle. For example there is org-contacts where nodes are meant to
be "People". Such development is so much hard coded.

Simpler would be to define the type of nodes and work by their
types. One could need just one type table and one node table and that
is about all.

The type table could say:

- this node is heading
- or, this node is text under heading
- or, this node is paragraph among many others
- or, this node is hyperlink to WWW URI
- or, this node is hyperlink to file
- this node is movie to play
- this type is PDF to be opened on page 3
- this one is really note
- this one is note but with action assigned like TODO, etc.

Nodes could have its properties defined like for anything. Nodes can
reference its parent nodes. Node can be parent to any heading.

Once such 2 tables are defined magic starts to take place, it becomes
its subtree with all kinds of node types including Babel execution and
similar. Meta data is meta, it can be updated but need not be
visible. Computer is handling meta data.

Node can be a page in a subtree that becomes a website.

Node can be object for person or company, or just anything.

I am currently using my nodes to quickly research various subjects by
using that type of dynamic knowledge repository.

Org file is dynamic knowledge repository.

About Dynamic Knowledge Repositories (DKR)
https://www.dougengelbart.org/content/view/190/163/

Then around 2015 I have discovered Cherrytree and have made bunch of
notes with it, and that is self-sufficient one program that keeps
simple things simple as it is much more simpler than Org. It offers a
visual interface to all features related to the knowledge management

Cherrytree - hierarchical note taking application with rich text and syntax highlighting
https://www.giuspen.com/cherrytree/

TAGS in Cherrytree are automatic if I remember well, TAG is simply
name of the node. If I call node TODO, then nodes under are simply
TODO items. 

Later I discovered there is something similar in Emacs so I started
with Org and use it mostly for instruction writing and project
management. And I find many options over kill for me. On the other
hand my usage of Org would be overkill for somebody, so it depends of
viewpoints.

In general I was always using headings and subheadings, trees, lists, notes by using text.

Somewhere 2004 I started using Markdown one among first as I found it
simpler than ASCIIDOC and M4, but not as perfect.

2020 and 2021 I like to raise the level of dynamic knowledge
journaling to the meta level where I think less about underlying
functions in software.

That experience also tells that Org is definitely not simple.

Among hnb, GeDaFe database approach, Cherrytree and Org mode, for
"keeping things simple" like note taking and TODO items, project
management, Cherrytree was the best for me.

Among all those for keeping complex things simple the database
approach is the best. Of course that I use database within Emacs and
it is not visible to user what it is. Database allows simultaneous
operation by several people on distance and that is the groupware
feature as envisioned by Doug Engelbart.

For document writing and nice formatting with LaTeX export, Org mode
is the best personally.

For notes, database based notes are best as only so I have relations
between the note and other objects. With 200,000 contacts in the
database which I can quickly access and assign notes to them, how
would that work with Org? Hardly. Notes are related to people, to
projects, plans, opportunities, research subjects, companies and so
on. There is no simple way in Org mode to relate one note to multiple
other related subjects.

And people try to do exactly that, people are developing Org in the
manner to relate objects in Org file to anything else. And they do
hard work as they do it manually. Relational database speeds up and
does not tell user to manually hyperlink various relations.

I have sent thousands of tasks to people by using this function
below. And I had to define for each Org file who are "assignees" for
that Org file. And I have like 50 assignees, so I need to copy and
paste their nick names or identifiers into the Org file. There it
comes the attribute of being "self-sufficient", it becomes
self-sufficient because I had to define all assignees into that
specific file, but I find it tedious and not useful.

(defun rcd/org-send-assigned-task ()
  "Sends assigned task to designated individual as Org"
  (interactive)
  (let* ((member-data (rcd-org-extract-assigned-member-email-data))
	 (id (if member-data (first member-data) nil))
	 (signature (if (equal (type-of (symbol-value (fifth member-data))) 'cons)
	 		(third (symbol-value (fifth member-data))) ""))
	 (file (rcd-org-subtree-to-file signature))
	 (subject (rcd/org-find-headline))
	 (esubject (escape-% subject))
	 (ask (unless id (y-or-n-p "No assignment found. Do you want to send it by email?")))
	 (name (if id (third member-data)))
	 ;; (name (if ask (read-from-minibuffer "Name:") name))
	 (voice (format "The task '%s' is being sent to '%s'" subject name))
	 (email (if id (if (equal (type-of (fourth member-data)) 'cons)
			   (car (fourth member-data))
			 (fourth member-data))))
	 (email (if ask (cf-search-email (read-from-minibuffer "Search for email: ")) email))
	 (really (y-or-n-p (format "Do you really want to send it to: %s?" (if ask email name)))))
    (if (and really (or id ask))
      (if (string-match "@" email)
	(progn
	  ;; (message (escape-% subject))
	  (speak voice)
	  (mutt-send-file name email esubject file))
	(message "No email specified"))
      (message "Aborted sending."))))

> Moreover, some devs are even reluctant to hide metadata (like unique
> ID implemented in org-id module) from users (which is possible and
> also partially implemented).

I hope that I have demonstrated that one who develops could review
several various paradigms and learn more. I am fine with any decision
of design for Org mode as I use it as it is and I have for me other
ways of managing data. I am not sure how much those features have been
discussed to say that hiding meta data is good or not good. It is
better to discuss and find what is more useful.

What I can see is that people complain for speed and they build
extensions that help with it. Extensions are external while built-in
Org does not keep up with the dynamic how people expect it to be.

For example I would expect Org to be very simple, very very simple, I
would rewind it back to its roots. Somebody else would like
complexities. Currently I can see that Org is not made for
complexities. It is better to unwind the development and make Org in
core very basic and speedy and let people enhance it with external
packages. In general it is better to remain simple. Even that may not
be possible any more.

I see hard work by many people who try to enhance Org as hierarchical
knowledge of data because people want to have feature X, but group of
those enhancements in reality belong to relational databases and not
to text files. Developers wish to accommodate various people and yet
by doing so they develop it into complex software. (129 .el packages
for one Emacs mode!?)

Among those one shall read the Doug Engelbart's paradigms, then
especially if one is developer of system of data management that many
people use, one shall explore other paradigms, including various
approaches to data handling. And overall one shall keep it simple as
it is main fundament of Org to be simple, while practical fact remains
that it is not anymore simple, not at all. 

I remember back in time with BASIC programming language that it had
also something like a built-in database that one could put on the
bottom of the file. Then there is also Perl's __DATA__ that is placed
straight into the file and one could also place images and other
stuff. Maybe the meta data could be kept this way in just one heading
named Meta data, and this heading would not be exported, it could be
hidden from the user and it could contain the database necessary for
single Org file. By pressing key to show properties one could see
properties or simply make them hidden. By making a copy of subtree the
metadata would also copy as usual. By exporting, one could make Org
without meta data, and I like using this information as I like sending
Org headings without personal meta data to third parties.