One vs many directories

One vs many directories

[Texas Cyberthal]

Having a tall directory tree with many leaves and branches is against
Org's philosophy.

Here is my argument that such a structure is objectively correct for
personal info management:

https://github.com/cyberthal/10-Bins-template

For the record, Org works fine with this, although I had to do a bit
of config to get around the default philosophy.

Re: One vs many directories

[Ihor Radchenko]

> Having a tall directory tree with many leaves and branches is against
> Org's philosophy.

I am wondering what you mean by Org's philosophy. Why would it have
anything to do with directories?

> Here is my argument that such a structure is objectively correct for
> personal info management:
>
> https://github.com/cyberthal/10-Bins-template

If you wanted feedback on this, I believe that Karl Voit's article below
is relevant. I guess you can even directly contact him if you wish - he
knows a lot about theory of classification.

https://karl-voit.at/2020/01/25/avoid-complex-folder-hierarchies/

Best,
Ihor

Texas Cyberthal <texas.cyberthal@gmail.com> writes:

> Having a tall directory tree with many leaves and branches is against
> Org's philosophy.
>
> Here is my argument that such a structure is objectively correct for
> personal info management:
>
> https://github.com/cyberthal/10-Bins-template
>
> For the record, Org works fine with this, although I had to do a bit
> of config to get around the default philosophy.

Re: One vs many directories

[Jean Louis]

* Ihor Radchenko <yantar92@gmail.com> [2020-11-21 08:15]:
> > Having a tall directory tree with many leaves and branches is against
> > Org's philosophy.
> 
> I am wondering what you mean by Org's philosophy. Why would it have
> anything to do with directories?

Texas will answer on that.

I am just intruding in the conversation.

From Org manual:

"Org keeps simple things simple."

I do think that Org does more than keeping things simple. When it
comes to seriously complex things Org is not there for such.

When there are more than 2000 people related notes, tasks,
calculations, questions arise if such better be kept in one Org file
or multiple Org files in one directory or multiple directories for
multiple Org files?!

With increasing complexities one can see that Org as such is good for
those purposes as written in the manual.

Org mode is not appropriate for meta-level organizing.

The Org paradigm is good for meta-level organizing.

That is where directories come into play.

In my opinion directories should never bother user. User should just
pre-define sets of directories such as: People, Groups, you name it,
and files should be accessible in such directories automatically.

I want Joe Doe's information, I do not need to know where it is stored
to browse to his information. I should be able to say to computer:

"Give me files of Joe Doe" and I would get it. It is 21st century and
we are not closely organized to how it should be.

Org mode paradigm shows that hierarchical tree is useful.

File system is a hierarchical tree.

But people who start working with Org mode will have better
hierarchical system on their file system, then those who don't work
with hierarchical tree of notes and other things.

It means Org mode and its paradigm is teaching people to organize
hierarchically.

But it does not offer meta-level of organization. Where such Org files
should be located? Again we think of hierarchical system, but when
things becomes really complex Org is not there for user as Org keeps
simple things simple. It does not keep complex things simple.

That is where systems of 10 Bins come in. For me it is type of
meta-level paradigm that keeps things organized.

My desire is that base Org package stops its development.

It should define itself and stop being developed due to its achieved
perfection as base system.

The idea is analogous to TeX and LaTeX where I consider TeX being
compact, perfect, finalized system without bugs (author awards people
to find bugs) and LaTeX and others are considered extensions.

TeX - typesetting system
https://www.tug.org/svn/texlive/

In the same way extensions could be added to Org but without base
system being each in a while tackled, changed, modified or
"upgraded".

> If you wanted feedback on this, I believe that Karl Voit's article below
> is relevant. I guess you can even directly contact him if you wish - he
> knows a lot about theory of classification.
> 
> https://karl-voit.at/2020/01/25/avoid-complex-folder-hierarchies/

Thank you for the reference, that is subject of my current
research.

Personally I do not find complex folder hierarchies as main
problem. Not at all.

I find that main problem is when each individual is faced with the
problem of organizing and cannot handle complex thing of
organizing. The trouble is exponential to the number of computer users
on the planet.

If we look at not complex organizing then file system organizing can
be easily left to the user for the user to decide on each filing how
and where files are organized. That is what majority of people on the
planet do.

When we come to somewhat raised or complex level of organizing then
one can see that if such organizing is left to the user is that file
system mostly becomes mess. That is where system as 10 Bins come into
play or any such similar organizing system.

For the simple organizing it is better that users is prepared for
complex organizing as time and number of files on the system
inevitably lead there. So it is generally better for population and
computing to switch to better paradigms of organizing
information. Those paradigms have been already designed by Doug
Engelbart and are followed by various software and concepts of
computing.

Doug Engelbart Institute - Boosting mankind's capability for coping with complex, urgent problems
https://www.dougengelbart.org/

On that meta-level of organizing user should rather define some groups
or subjects such as people, lists, groups, plans, movies, etc. Then
user should define various possible relations between each other. And
template for such relational file system would be given onto each
computer and could be configurable by every user.

Then when it comes to filing of the files user could say:

$ file-file this-file.org

User would answer few questions, like if to file by person, or by some
list, or movie subject, user could select to which movie genre that
movie belong. File would be filed and WHERE it is filed would be
irrelevant for the user. User could find the real path to the file,
but because it is not relevant for the user it would not be of much
importance.

When user wish to access the file, one would find it as:

$ find-file

in the next step user could decide:

- to search for file by using its name

- to search for file by subject, such as movie

- to search for file by genre or sub-group(s) of the subject

- to search by people, groups, lists, etc.

Then file would be found and it would not be relevant if the file is
located in x/x/x/x/x/x/x structure on the file system.

In general when user is searching for the file, when such file is
located file could be launched, edited, viewed or played, or shared to
other people.

How the file system is organized would be left to the algorithm on
computer.

There would be no need for user to browse the file system but in
exceptional cases.

Reference:
https://www.dougengelbart.org/content/view/110/460/#2b1a1

"   Every object that someone might validly want/need to cite or
   otherwise access should have an unambiguous address, capable of
   being portrayed in a human readable and interpretable manner. Most
   common existing spreadsheet programs have provisions similar to
   this for cell addressibility." 2b1a1

The referenced link demonstrates itself: 110/460/#2b1a1 is the
unambiguous address capable of being cited or referenced.

Both filing and finding of files would be kept as history. This is for
hyperlinking and referencing purposes.

After filing of a file, the history items can be used as a reference
for immediate classification. I have filed the file, but did not
designate the name of the file and other attributes. I need those
attributes for later referencing and searching.

$ file-file this-file.org

would automatically create a hyperlink to that file and hyperlink
could be kept in history. If there is some central dynamical knowledge
repository such hyperlink could be automatically inserted into the
database (DKR). 

About Dynamic Knowledge Repositories (DKR)
https://www.dougengelbart.org/content/view/190/163/

Side note: Org file is one type of dynamic knowledge repository.

Then that same hyperlink could be reused in many various Org files. Or
hyperlink in the Org file could just point to the DKR hyperlink, this
way the eventual change of location of the file would not affect many
Org files. All Org files would be hyperlinking to meta-hyperlink that
tells where the file is really located.

When searching for files by any manner, I should also be able to
quickly obtain hyperlinks, as maybe I wish to place such in the list
without me having to search for same file again.

After:

$ file-find

I would find the file but have hyperlink to the file ready in memory
or history to place it in Org file or any other file or database
collection. Or I should be able to share it with people.

I rather expect Org mode to have similar form as OPML so that every
atom or object in the Org file may get its reference.

Example is that I can easily hyperlink to headings but I cannot easily
hyperlink to specific list items under headings. And I cannot easily
link to paragraphs. This is because Org file structure is not finely
grained. I wish it could be so, but it isn't.

For this reason I am considering to convert some Org files that are
useful for referencing to personal structured meta-level dynamic
knowledge repository. org-id tries to handle these problems of
complexities but also makes Org files less readable.

Now back to reading Karl Voit's work.

Re: One vs many directories

[Texas Cyberthal]

Hi Jean,

I'll use some of the concepts in the first half of your email.  I
disagree with the second.

> In my opinion directories should never bother user. User should just pre-define sets of directories such as: People, Groups, you name it, and files should be accessible in such directories automatically.

Productivity studies show that navigation dominates search.  Human
animals are natural pathfinders and walking computer paths with
ergonomic file explorers such as Dired increases mastery of the
subject matter.

This value is trivial with retrieval tasks such as a person's name,
which is why 10 Bins stores such names in a flat list of directories,
sorted alphabetically by last name.  It is easy to integrate an
automated retrieval script with such a predictable path.

I think Treefactor is the correct means to refile files and
directories, not a CLI program.  The benefits of incremental inductive
refiling outweigh those of the system you described.  Incremental
inductive refiling isn't compatible with your suggested automation,
history and link features.  Proper repo management is also
incompatible.  For example, text and binary files should be handled
separately.

I document rapid iterative inductive refiling here:
https://treefactor-docs.nfshost.com

I've come to depend on its cognitive benefits.

Re: One vs many directories

[Marvin ‘quintus’ Gülker]

Am Samstag, dem 21. November 2020 schrieb Texas Cyberthal:
> Productivity studies show that navigation dominates search.  Human
> animals are natural pathfinders and walking computer paths with
> ergonomic file explorers such as Dired increases mastery of the
> subject matter.

This sounds interesting. As far as I am aware, search interfaces are
increasingly deployed instead of hierarchical structures (just take the
loss of the classic Windows Start Menu as an example). You suggest that
this is unergonomic, which I, from a personal taste, would tend to
agree. Do you have a source for the "productivity studies" you refer to?
I would like to skim through them and see if I can cite them the next
time someone wants me to convince to use a search-based interface.

  -quintus

-- 
Dipl.-Jur. M. Gülker | https://mg.guelker.eu |    For security:
Passau, Germany      | kontakt@guelker.eu    | () Avoid HTML e-mail
European Union       | PGP: see homepage     | /\ http://asciiribbon.org

Re: One vs many directories

[Jean Louis]

* Texas Cyberthal <texas.cyberthal@gmail.com> [2020-11-21 11:32]:
> Hi Jean,
> 
> I'll use some of the concepts in the first half of your email.  I
> disagree with the second.
> 
> > In my opinion directories should never bother user. User should just pre-define sets of directories such as: People, Groups, you name it, and files should be accessible in such directories automatically.
> 
> Productivity studies show that navigation dominates search.  Human
> animals are natural pathfinders and walking computer paths with
> ergonomic file explorers such as Dired increases mastery of the
> subject matter.

Do you mean that navigating file system dominates the search?

I also think that navigating file system dominates the search if that
is what you refer. I think that users are inclined to navigate because
computing tools such as file managers are given to users for
that. Users did not get better systems to find or file files or other
documents in computing. In other words I wish to say that we are under
developed.

Navigating does not necessarily contribute to production. Productivity
may say what it wants but it may not reach those who are actually more
productive without using the navigation. So studies may not tell us
what is more productive, such may only tell what is currently used
within the subject of being productive.

You mentioned 2 things, navigation and search. Since years I am
integrating pieces in my computing that drives me into direction of
neither navigating nor searching. Examples:

- I read your email in Mutt. Maybe I remember something you wrote
  earlier but not quite well and I wish to find your previous email. I
  click ESC-v and I can view all your previous emails. In this sense I
  do not need to know your email address and where your emails are
  stored on my system.

  For this particular type of object "Emails of user Texas" neither I
  am searching, neither navigating. I do not spend time searching and
  I do not spend time navigating to that object. It is by one click or
  two all in front of my eyes.

  Underlying functionality is very simple. All emails of users can be
  automatically (by sieve) or semi-automatically by key press saved
  into personalized mailboxes ~/Maildir/person@example.com and by
  clicking ESC-v in Mutt, email address is extracted from the email I
  was reading and I new Mutt instance opens with all emails from
  ~/Maildir/person@example.com -- then after reading, I click q and I
  am back in your first email, the top level email in the inbox.

  I could as well make it more automated, I could answer all emails
  and finish with Mutt, and upon finishing all emails could be sorted
  in personalized email files.

- example with files belonging to user, let us say hyperlinks or
  anything, any piece of information, I could just press F4 on email
  and I could access all information related to that user. I do not
  need to know the phone number and I cand send SMS or initiate the
  call, share the contact, send email or fax, see all pictures of this
  person, notes, tasks, and financial transactions.

  I neither navigate neither search there. Maybe better way to call
  this type of locating objects is relational accessing. Somebody may
  correct me.

  There is some object like contact named "Texas Cyber". If object has
  any relation to anything else, then anything else can be displayed
  and found right there.

  Instead of me searching, computer is searching.

  Instead of me navigating, computer is navigating.

> This value is trivial with retrieval tasks such as a person's name,
> which is why 10 Bins stores such names in a flat list of
> directories, sorted alphabetically by last name.  It is easy to
> integrate an automated retrieval script with such a predictable
> path.

Take care of duplicates. When marketing contact database is growing
fast, some times 1000 people per day or more. People have same
names. Often one cannot even know what is first and what is last
name. You may know it for your country, in other countries is not
so. Then those people engage in a course on distance. They are sending
me images and files as results of their course assignments. I have to
file the files in proper folder. Because names are not always unique I
better file it under unique IDs, and keep separate symlinks with names
of people to such unique IDs whereby symlinks can be automatically
updated.

That alone makes things easily very simple at least for objects such
as people because people's names are in the file system.

Then simple `locate` command can be used to find Texas Cyber's files:

#!/bin/bash
locate -e -d /home/data1/protected/.locate.database -A -i $@

I think that switch -A helps in finding all of these:

Texas Cy

Cyber Tex

or similar variations.

But in general I would just click ESC-e and I would get your profile
and access to all files because location by email address from an
email document is pretty much decisive.

If person is not there, person is then created in the database.

Re: One vs many directories

[Texas Cyberthal]

Hi Jean,

> Navigating does not necessarily contribute to production. Productivity may say what it wants but it may not reach those who are actually more productive without using the navigation. So studies may not tell us what is more productive, such may only tell what is currently used within the subject of being productive.

Outside 10 Bins, navigation is often negatively productive.  Whenever
the user navigates bad tree structure without correcting it on the
fly, he suffers profitless friction.  That's why Treefactor combines
with JiT sorting to make navigation and sorting a single activity.

Frankly I was surprised people prefer navigation despite being
generally so bad at tree structure.  In the absence of good structure
and tools, search is much better.

I agree, email should be database-centric.  Manually organizing emails
into folders (or worse, trees) is therefore wrong except in tiny
doses.

> Take care of duplicates. When marketing contact database is growing fast, some times 1000 people per day or more. People have same names. Often one cannot even know what is first and what is last name. You may know it for your country, in other countries is not so. Then those people engage in a course on distance. They are sending me images and files as results of their course assignments. I have to file the files in proper folder. Because names are not always unique I better file it under unique IDs, and keep separate symlinks with names of people to such unique IDs whereby symlinks can be automatically updated.

This is clearly a CRM database use case.  In this situation, the CRM
should define the unique ID, and then Textmind should accept it.  You
can still use the directory tree, though.  Just file it under
~/Surname-given-name/ID-number/~ for the non-unique name.  Recursively
searching ~/1-Textmind/2-Linked/7-Name/2-Flat/~ for a directory named
~/ID-number/~ will find the target even if his name changes slightly.
So you can save time by not inputting every scrap of text and files
into the CRM.

Re: One vs many directories

[Jean Louis]

* Texas Cyberthal <texas.cyberthal@gmail.com> [2020-11-21 18:01]:
> Hi Jean,
> 
> > Navigating does not necessarily contribute to production. Productivity may say what it wants but it may not reach those who are actually more productive without using the navigation. So studies may not tell us what is more productive, such may only tell what is currently used within the subject of being productive.
> 
> Outside 10 Bins, navigation is often negatively productive.  Whenever
> the user navigates bad tree structure without correcting it on the
> fly, he suffers profitless friction.  That's why Treefactor combines
> with JiT sorting to make navigation and sorting a single activity.
> 
> Frankly I was surprised people prefer navigation despite being
> generally so bad at tree structure.  In the absence of good structure
> and tools, search is much better.

Do you really think people prefer? Or they simple have no other
option?

If I have no other option to drink a juice I will drink water, not
necessarily I prefer drinking water in hot sunshine. I like
Apfelschörle.

Searching file contents is database search. I am not any more fan of
that. Tools like Beagle or Tracker are making basically double
database of files only for me to find or search files. Most of time I
am only searching for meta data, not for contents. With 1000 GB one
would need to have maybe that much indexed database and constant
updating of files and their positions.

That is why I prefer relational pinpointing or relational access and
actions or locating files by using indexes.

Relational access would mean when you inspect the object to quickly
jump to all relational pieces of information relating to that
object. If I look on person's name there need not be any note or
contact information but I should be able to quickly access such
information. And each object should have general actions like actions
relating to email object could be to send email, delete email,
forward, etc. that is what mail readers do. Action on file relating to
user should be to quickly see emails of the user, social networks,
websites, to call user, SMS, send information, jump into XMPP chat,
share the file or request new version of file and similar.

Locating files by indexes would be to index only meta data and then to
use searches to find meta data such as title, date, author, or various
attributes. Tools like locate and similar do that. 

> I agree, email should be database-centric.  Manually organizing
> emails into folders (or worse, trees) is therefore wrong except in
> tiny doses.

You missed this:

- I am reading email, answering and if I wish to keep it all I do is
  `s` to save it into the mailbox related to the email address:

  ~/Maildir/texas@example.com

  Emails that I send to user are saved to same mailbox.

  That is all. No thinking, nothing, saving goes automatic. This
  single plan of filing emails enables me to see all conversations
  related to that email address.

  Database keeps all email addresses of specific person

  $ emailsof texas@examp

  would give me 3 views if there are 3 email addresses, I could
  basically review all conversations of that person.

  There need not be any true database like `mu` or `notmuch` as this
  way I find most of times what I need. I can search inside of
  mailboxes easily.

  I would not keep emails in the database like PostgreSQL, no. Emails
  have to be accessed by mail readers and there are just few that
  would be supporting databases. 

> > Take care of duplicates. When marketing contact database is
> > growing fast, some times 1000 people per day or more. People have
> > same names. Often one cannot even know what is first and what is
> > last name. You may know it for your country, in other countries is
> > not so. Then those people engage in a course on distance. They are
> > sending me images and files as results of their course
> > assignments. I have to file the files in proper folder. Because
> > names are not always unique I better file it under unique IDs, and
> > keep separate symlinks with names of people to such unique IDs
> > whereby symlinks can be automatically updated.
> 
> This is clearly a CRM database use case.  In this situation, the CRM
> should define the unique ID, and then Textmind should accept it.  You
> can still use the directory tree, though.  Just file it under
> ~/Surname-given-name/ID-number/~ for the non-unique name.  Recursively
> searching ~/1-Textmind/2-Linked/7-Name/2-Flat/~ for a directory named
> ~/ID-number/~ will find the target even if his name changes slightly.
> So you can save time by not inputting every scrap of text and files
> into the CRM.

That is right, good idea to file under surname/ID. I would rather
prefer the approach ID/Full-Name as if directory is automatically
created by its ID, so I do not need to think about it.

CRM is for me not the database, it is method of management of anything
related to people and I call it Central Files. I do not put text files
into database as that converts them into something else, they are not
text files. That is nonsense invented by people who try to use
exclusively web interface to manage anything and such interface is
limiting user. User for example is unable launch `mpv` video player on
the video file belonging to the person Joe Doe as browser will not
allow it to launch external programs.

CRM or Customer Relationship Management is just way of thinking, it
does not need to be computerized. Rolodex was one way of CRM method
and it worked well and works today well. Larger organizations have and
still have paper files, they open paper file and can see previous
conversation, which officer or sales manager handled the person and
how, and they would make a call and put note in the file and file it
back in place. To handle daily 100 or 300 people is quite possible by
using this system. I was handling usually 800 files per week and wrote
this many personal letters. Just take the bunch of paper files and go
over files, open, call, invite customer, make a note on paper, put
note in the file, close it. It is pleasure working that way and I can
tell out of experience. This is for reason that files are still
physical object that staff handling it can keep it in the
hands. Collaboration is also there, multiple staff members simply
exchange such paper files between themselves. And multiple staff
members can look into such files and see what other staff member has
communicated, offered, sold to specific person. And I still do have
paper files as such are also part of digital management. It is
often illegal not to have paper files pertaining to various
entities. If there are contracts, I have to have paper files, it is
yet another object. On paper file there must be unique ID written down
by hand. Contracts may be in such paper file. Digital index must tell
where the paper file is sorted, maybe it is A2 box on the shelf in the
green room.

Re: One vs many directories

[Dr. Arne Babenhauserheide]

[-- Attachment #1: Type: text/plain, Size: 1096 bytes --]


Jean Louis <bugs@gnu.support> writes:

> When there are more than 2000 people related notes, tasks,
> calculations, questions arise if such better be kept in one Org file
> or multiple Org files in one directory or multiple directories for
> multiple Org files?!

This came up multiple times in discussions. I think that it is wrong,
and the reason comes down to efficiency. If you want to work
efficiently, then sub-second delays matter, and having 4 files with
500 entries means that to search in them you need to open 4 files.

If you have 100 files with 20 notes each, you have to open 100 files.

Almost any other computing cost pales compared to opening files.

My current setup has around 1200 notes in 10 files (most of them in the
two main files, some of the notes are several pages long, but most take
up around half a page). Using org-rifle
(https://github.com/alphapapa/org-rifle) I can full-text-search them
with barely perceptible delay on a system clocked down to 1 GHz.

Best wishes,
Arne
-- 
Unpolitisch sein
heißt politisch sein
ohne es zu merken

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 1125 bytes --]

Re: One vs many directories

[Texas Cyberthal]

Hi Arne,

*Almost* any computing cost pales, but not the computing cost of Emacs
choking on rendering large files.

Doubtless there are ways to mitigate that issue.  I'm unsure what the
tradeoffs would be.  Perhaps my Spacemacs does too much prettifying of
my Org buffers.  But I like pretty buffers.  It's easier to just stay
under 1 MB file size.

File opening cost should be reduced by storing text files on an SSD.
A small one will do.

Textmind/Exomind buffers should be left open.  Thus the opening cost
is paid once per session per needed file.  That is acceptable.

I'm not sure what kind of lagless Org database operations are required
in your workflow, but I suspect they could be mostly replaced by a
proper Textmind workflow and 10 Bins structure.

The alternative is to do everything in a few big files.  Org has too
much potential to misfold or otherwise mangle huge outlines in ways
difficult to spot and repair.  Nor do I want to squint at endless
asterisks when I could have Dired ergonomics instead.

Grepping my 94 Mb 6562 files (excluding archive) Textmind for
"elephantine" takes a few seconds, which is fine.  Org searching for a
nonexistent UID link takes a few minutes, which is why I run that
search nightly, to refresh the index.  My Org Agenda search scope is
only 252k in 58 files and is effectively lagless.

I guess I avoid the problem you're talking about by mostly excluding
bulk prose from the Agenda directory.  They're fundamentally different
and should be handled differently.  One is about human language, the
other is about database metadata.  The temptation to do everything
inline just because Org supports it is one of Org's biggest traps.
It's like the trap of trying to outline strictly when you should be
rambling.

Re: One vs many directories

[Jean Louis]

* Texas Cyberthal <texas.cyberthal@gmail.com> [2020-11-21 18:46]:
> I guess I avoid the problem you're talking about by mostly excluding
> bulk prose from the Agenda directory.  They're fundamentally different
> and should be handled differently.

Well said. 

> One is about human language, the other is about database metadata.

That is so.

In fact the properties, custom ID and else inside is mostly visually
disturbing me though it is necessary. Before knowing about Org I have
already had a system of planning, projects, tasks, and all that could
be related to people, groups and what other else things. This part of
Org that is dependant on meta data is better managed through
dedicated databases. 

> The temptation to do everything inline just because Org supports it
> is one of Org's biggest traps.  It's like the trap of trying to
> outline strictly when you should be rambling.

Temptation is brought through the usage of Org as there are not so
many choices beyond it. Org is meant to keep simple things
simple. Data being managed may easily grow into complex thing.

Re: One vs many directories

[Texas Cyberthal]

Hi Jean,

> That is good and isn't it general way of sorting things? I guess that general computer users may not be aware that they could make nice hierarchical tree of directories.

It's not that they're unaware.  Everybody with a mouse and Windows
Explorer tries to make good directories.  It's just that
Dired+Treefactor's order of magnitude improvement in speed and fluency
means that the directory tree can be mind-synced as never before,
making walking the tree an education in itself.  With so much
intelligence embedded in each level, bypassing it makes little sense.
Also one should check during the walk whether key info is waiting in
transit for batch refiling.

For classic database tasks, I'd rather use Postgres than symlinks.  I
already use symlinks enough for conveniences such as connecting
synonyms.  Too many symlinks make paths unpredictably brittle,
chilling sync dynamism, creating rigidity.

Textmind is documented at http://cyberthal-docs.nfshost.com

I neglected Dr. Arne's honorific.

Re: One vs many directories

[Jean Louis]

* Texas Cyberthal <texas.cyberthal@gmail.com> [2020-11-21 21:02]:
> Hi Jean,
> 
> > That is good and isn't it general way of sorting things? I guess
> > that general computer users may not be aware that they could make
> > nice hierarchical tree of directories.
> 
> It's not that they're unaware.  Everybody with a mouse and Windows
> Explorer tries to make good directories.

People try and people fail. I know hackers only on distance, I do not
know them personally face to face. Often computer users that I see in
offices like stationary, even sometimes legal office, have their
desktops overloaded of files on top of files on top of files where
majority of them are named something like unknown or whatever new-file
and similar nonsense. All sorting is taking place under Desktop on
Desktop space and it becomes mess.

> It's just that Dired+Treefactor's order of magnitude improvement in
> speed and fluency means that the directory tree can be mind-synced

Good term, mind synced. That is how it should be. In my opinion
various paradigms of file sorting should be sorted in a list similar
to those Awesome lists. Sorting files how mind thinks is how it should
be. Computer should help user to complement the mind and help in
execution of actions and not bother the mind or make mind confused
what is really the case with majority of users.

Side note: I was always sorting files, people, stuff from the command
line or from web browser into their places. And I mostly used readline
in bash for completion or finding of the selection candidate.

In Emacs I am using `ivy-mode` or `helm`, `selectrum` and similar,
Emacs offers quite good interface for completions.

And often I am completing file locations, maybe I wish to file into
this or the other subject. So the list of subjects has to be brought
up.

In X and outside of Emacs there is dmenu and I could use it even from
Emacs inside out. 

(defun dmenu-completing-read (prompt collection)
  ;; &optional predicate require-match initial-input history def inherit-input-method)
  "Uses external `dmenu' command for Emacs completions."
  (let* ((collection (concat (string-join collection "\n") "\n"))
	 (file (string-to-file-force collection "/dev/shm/collection"))
	 (dmenu "dmenu"))
    (with-temp-buffer
      (call-process dmenu file (current-buffer) nil "-fn" "DejaVu:pixelsize=30" "-l" "10" "-i" "-b" "-p" prompt "-nb" "dark goldenrod" "-nb" "black")
      (string-trim (buffer-string)))))

(dmenu-completing-read "Subject: " '("People" "Work" "Group"))

I find dmenu as excellent tool to provide it for functions that are used within Emacs and that may be used outside of Emacs to complement Emacs filing of files.

For example Rox filer file manager or Nautilus can invoke external command that files the file into specific directory by using dmenu. Choose the directory by few words, and if directory is semnatically written, file may be filed easily.

Three interfaces for selection among large line-based items were needed:

- [X] Emacs
- [X] X interface: Dmenu
- [ ] Console. I was using readline with TAB, it is not visual enough

Now I found finally `fzf`.

$ mv courier-crm114 `find Work/Computer -type d | fzf`

Then in the next step I can type "mail" or "spam" subdirectories and file it there.

fzf is great tool for console and terminal emulators. It similar
things on console what helm does on Emacs and dmenu on X.

- [X] Console: fzf

fzf is great tool that may be used for finding stuff, filing, retrieval of stuff, launching or opening files. Example:

$ mimeopen `locate *.org | fzf`

and similar with dmenu:

$ mimeopen `locate -d .locate.database  -A Documents .org | dmenu -fn DejaVu:pixelsize=20 -l 10 -i -b -p "Org: " -nb "dark goldenrod" -nb black `

As my files are in ~/Documents/Org/ I would find at least 185
files ending with .org there by its name and quickly open it with
emacs. Instead of "mimeopen" I could as well use emacsclient.

If you have set your mimeopen to open it by Emacs, you will
quickly locate the file and open it. Provided that Org file names
have some built-in semantics.

Re: One vs many directories

[Ihor Radchenko]

> In fact the properties, custom ID and else inside is mostly visually
> disturbing me though it is necessary.

FYI: org-custom-properties and org-toggle-custom-properties-visibility


Jean Louis <bugs@gnu.support> writes:

> * Texas Cyberthal <texas.cyberthal@gmail.com> [2020-11-21 18:46]:
>> I guess I avoid the problem you're talking about by mostly excluding
>> bulk prose from the Agenda directory.  They're fundamentally different
>> and should be handled differently.
>
> Well said. 
>
>> One is about human language, the other is about database metadata.
>
> That is so.
>
> In fact the properties, custom ID and else inside is mostly visually
> disturbing me though it is necessary. Before knowing about Org I have
> already had a system of planning, projects, tasks, and all that could
> be related to people, groups and what other else things. This part of
> Org that is dependant on meta data is better managed through
> dedicated databases. 
>
>> The temptation to do everything inline just because Org supports it
>> is one of Org's biggest traps.  It's like the trap of trying to
>> outline strictly when you should be rambling.
>
> Temptation is brought through the usage of Org as there are not so
> many choices beyond it. Org is meant to keep simple things
> simple. Data being managed may easily grow into complex thing.

Re: One vs many directories

[Jean Louis]

* Ihor Radchenko <yantar92@gmail.com> [2020-11-22 09:37]:
> > In fact the properties, custom ID and else inside is mostly visually
> > disturbing me though it is necessary.
> 
> FYI: org-custom-properties and
> org-toggle-custom-properties-visibility

***** Full contact information is required
      :PROPERTIES:
      :CREATED:  [2018-10-08 Mon 21:34]
      :ID:       06781e66-0382-4833-a61e-0d76e317593f
      :END:

Thank you. Am I supposed to declare these properties in
org-custom-properties for it not to be visible?

Re: One vs many directories

[Ihor Radchenko]

> ***** Full contact information is required
>       :PROPERTIES:
>       :CREATED:  [2018-10-08 Mon 21:34]
>       :ID:       06781e66-0382-4833-a61e-0d76e317593f
>       :END:
>
> Thank you. Am I supposed to declare these properties in
> org-custom-properties for it not to be visible?

Yes. I use

(setq org-custom-properties '("ID" "STYLE" "REPEAT_TO_STATE" "CREATED"))

Then, you will need to run M-x org-toggle-custom-properties-visibility
or add it to org-mode-hook.

Your example will then look like 

***** Full contact information is required
      :PROPERTIES:
      :END:

Beware of using this on large org files. The command creates a huge
number of overlays (one overlay per property line), which may slow down
Emacs.

Hiding the emptied property drawer is currently not possible, unless you
use my WIP patch [1]. The branch also mitigates the issue with large org
files. However, this particular functionality was opposed by other devs
earlier. Further discussion will follow once more important parts of the
patch are resolved.

If you use the patch, you can also set
org-custom-properties-hide-emptied-drawers to 't. Then, your example
should look like

***** Full contact information is required

[1] https://orgmode.org/list/87lfh5vvrp.fsf@localhost/

Best,
Ihor

Jean Louis <bugs@gnu.support> writes:

> * Ihor Radchenko <yantar92@gmail.com> [2020-11-22 09:37]:
>> > In fact the properties, custom ID and else inside is mostly visually
>> > disturbing me though it is necessary.
>> 
>> FYI: org-custom-properties and
>> org-toggle-custom-properties-visibility
>
> ***** Full contact information is required
>       :PROPERTIES:
>       :CREATED:  [2018-10-08 Mon 21:34]
>       :ID:       06781e66-0382-4833-a61e-0d76e317593f
>       :END:
>
> Thank you. Am I supposed to declare these properties in
> org-custom-properties for it not to be visible?

Re: One vs many directories

[Jean Louis]

* Ihor Radchenko <yantar92@gmail.com> [2020-11-22 11:33]:
> > ***** Full contact information is required
> >       :PROPERTIES:
> >       :CREATED:  [2018-10-08 Mon 21:34]
> >       :ID:       06781e66-0382-4833-a61e-0d76e317593f
> >       :END:
> >
> > Thank you. Am I supposed to declare these properties in
> > org-custom-properties for it not to be visible?
> 
> Yes. I use
> 
> (setq org-custom-properties '("ID" "STYLE" "REPEAT_TO_STATE"
> "CREATED"))

I've tried then :PROPERTIES and :END: still remain. It is possible
even not to show PROPERTIES but :END remains.

> Then, you will need to run M-x org-toggle-custom-properties-visibility
> or add it to org-mode-hook.

Thank you, I will rather give up on that. It is not designed for it.

> Hiding the emptied property drawer is currently not possible, unless you
> use my WIP patch [1]. The branch also mitigates the issue with large org
> files. However, this particular functionality was opposed by other devs
> earlier. Further discussion will follow once more important parts of the
> patch are resolved.
> 
> If you use the patch, you can also set
> org-custom-properties-hide-emptied-drawers to 't. Then, your example
> should look like
> 
> ***** Full contact information is required

Thank you, I will choose to give up on that as it has not get much
priority and maybe I stop creating unique IDs.

I will explore how to re-file tasks into the SQL database.

Re: One vs many directories

[Dr. Arne Babenhauserheide]

[-- Attachment #1: Type: text/plain, Size: 2015 bytes --]

Hi Texas,

> Grepping my 94 Mb 6562 files (excluding archive) Textmind for
> "elephantine" takes a few seconds, which is fine.  

For the sake of ruining my argument ( :-) ), you might want to check ripgrep.

Searching within 30k files of in total around 150 MiB for
ProviderBuilderFactory (guess what type the files are :-) ) takes 0.4s
with ripgrep.

(that’s on an nvme (M.2) disk)

It’s still too slow for interactive use.

Within 1k files of in tootal 7 MiB it is fast enough.

> Org searching for a nonexistent UID link takes a few minutes, which is
> why I run that search nightly, to refresh the index. My Org Agenda
> search scope is only 252k in 58 files and is effectively lagless.

That lagless is what I see as being required for actual operation.

> I'm not sure what kind of lagless Org database operations are required
> in your workflow, but I suspect they could be mostly replaced by a
> proper Textmind workflow and 10 Bins structure.

I need instant search in the knowledge database and quick filing of
tasks. Also I need the agenda to create a clocktable (that’s on the
limit of being too slow) and the calendar and tasks of the week.

Also I need quick filing of notes and quotes (in specific files, not
part of the agenda) and of long-form articles, one file per article
(using journal.el, also outside the agenda, searched using M-x deft),
and quick creation of website entries for a given category within the
site (i.e. M-x draketo-software).

> I guess I avoid the problem you're talking about by mostly excluding
> bulk prose from the Agenda directory.  They're fundamentally different
> and should be handled differently.

I do that, too. One is source code, one is organisation tasks. I link
from org into the source code, though (but never check the targets).

I do use org-planning within prose, but there the scope is only the one
org-document.

Best wishes,
Arne
-- 
Unpolitisch sein
heißt politisch sein
ohne es zu merken

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 1125 bytes --]

Re: One vs many directories

[Texas Cyberthal]

Hi Dr. Arne,

> The only part that hits performance limits is the agenda.

Well, IIRC your Org Textmind is much smaller than mine.

> My current guess is that the agenta is slow because it has to parse all my 7500 clock entries, and it has to check the Todo states of around 1200 headings.

Ouch.  I'd rather keep a "ramble log" so I can reconstruct an exactly
honest time accounting, with discounts for partial attention, without
worrying about fiddly clockin/outs.  At least when working from home.
If clocking into a work site, that's different, because one can
reasonably bill for the entire time, with minimal clock toggling.

> Did you check against filesystem limits? At 10k entries in a directory typical filesystems start becoming slow. That’s the main reason I see for adding hierarchies.

10k entries in a directory sounds inhumanely unergonomic.  I guess my
biggest flat name directory might eventually reach that size?  In
which case I could just split it in the middle of the alphabet, or
similar solution.

Right now it's only 600.  If I guess a generous growth rate of 2 per
day, times 30 years, that would be an additional 22k.  Sounds
manageable.

Remember there are ways to consolidate entries even in flat "solid
names" directory.  It's advantageous to do so to facilitate isearch
matching.  For example, everyone with the same last name is one
directory.  Ditto everything that starts with the same word or even
prefix.  For example I have a directory called ~Wiki-~ and another
called ~Tru-~ which contains truth, Trudeau and Trump.

Most adults know 20-35k words.  That's not the same as "solid names"
known, but gives a ballpark on human memory size for a similar name
type.  I suspect computers will advance faster than anyone's Textmind
reaches the Dired lag limit.

No, if we are talking about scaling limits, then limits such as buffer
size and Agenda search speed are orders of magnitude more relevant.
Which problems deep tree nesting fixes.

A 10k entry directory is getting into enterprise territory, and I'm
sure enterprise has tech tricks that become worthwhile at that scale.

> There are scaling problems in every direction: Too many files per directory, too large files, too much content per heading, too many headings.

There are scaling problems from too much deep tree nesting, namely too
much fiddly ambiguous manual refiling.  Solution is flat "solid name"
directories just below feasible 10 Bins.  Work fine.

> I would have to build lots of additional tooling to make that work as well. Many of the tools in Emacs work better on large files than on many files — I will switch to more files when performance on large files reaches its limits.

Nah, my 100 mb (non archived) Textmind works fine.  I just separated
Agenda metadata from bulk prose.

I am curious how many headings I have, how would I count that recursively?

On Sun, Nov 22, 2020 at 8:04 PM Dr. Arne Babenhauserheide
<arne_bab@web.de> wrote:
>
>
> Texas Cyberthal <texas.cyberthal@gmail.com> writes:
>
> >> I need instant search in the knowledge database and quick filing of tasks. Also I need the agenda to create a clocktable (that’s on the limit of being too slow) and the calendar and tasks of the week.
> >
> >> Also I need quick filing of notes and quotes (in specific files, not part of the agenda) and of long-form articles, one file per article (using journal.el, also outside the agenda, searched using M-x deft), and quick creation of website entries for a given category within the site (i.e. M-x draketo-software).
> >
> > So your Org usage style quickly hits critical performance problems at scale.
>
> The only part that hits performance limits is the agenda. All the rest
> scales nicely. My current guess is that the agenta is slow because it
> has to parse all my 7500 clock entries, and it has to check the TODO
> states of around 1200 headings. Having multiple files would only add to
> that.
>
> > I don't have these problems.  Treefactor refiling is immune to scale.
>
> Did you check against filesystem limits? At 10k entries in a directory
> typical filesystems start becoming slow. That’s the main reason I see
> for adding hierarchies.
>
> > Org's many tools and tricks are still handy in niche cases, but they
> > don't cause scaling problems because they don't handle bulk info
> > management.  For example Org's refile tools are useful when writing
> > advanced documentation with large single-file outlines.  Most info
> > doesn't require that much organization.  It works fine as flat lists
> > of headings in a detailed directory tree.
>
> Or as sub-headings in a large outline.
>
> There are scaling problems in every direction: Too many files per
> directory, too large files, too much content per heading, too many
> headings.
>
> I would have to build lots of additional tooling to make that work as
> well. Many of the tools in Emacs work better on large files than on many
> files — I will switch to more files when performance on large files
> reaches its limits.
>
> I have one file where I’m reaching the limit. That’ my 7.3 MiB
> emacs-remember-mode.org file where I throw long-form articles for
> full-text search. I am considering to switch to a multi-file approach
> for that and then to use deft to retrieve articles.
>
> Best wishes,
> Arne
> --
> Unpolitisch sein
> heißt politisch sein
> ohne es zu merken

Re: One vs many directories

[Jean Louis]

* Texas Cyberthal <texas.cyberthal@gmail.com> [2020-11-23 12:51]:
> Hi Dr. Arne,
> 
> > The only part that hits performance limits is the agenda.
> 
> Well, IIRC your Org Textmind is much smaller than mine.
> 
> > My current guess is that the agenta is slow because it has to parse all my 7500 clock entries, and it has to check the Todo states of around 1200 headings.
> 
> Ouch.  I'd rather keep a "ramble log" so I can reconstruct an exactly
> honest time accounting, with discounts for partial attention, without
> worrying about fiddly clockin/outs.  At least when working from home.
> If clocking into a work site, that's different, because one can
> reasonably bill for the entire time, with minimal clock toggling.
> 

> > Did you check against filesystem limits? At 10k entries in a
> directory typical filesystems start becoming slow. That's the main
> reason I see for adding hierarchies.

From ext4 manual:

 dir_index
  Use hashed  b-trees to speed  up name lookups  in large
  directories.   This feature  is supported  by ext3  and
  ext4 file systems, and is ignored by ext2 file systems.

 dir_nlink
  This ext4 feature allows more than 65000 subdirectories
  per directory.

I think that file systems should be unlimited and fast in relation to
that. I have ~/Maildir with over 50000 subdirectories, direct access
is very easy and fast while listing takes some time.

If file system does not allow fast access it is time to replace it
with one that does allow it.

Now I wonder of HAMMER in DragonflyBSD is also slow with 50000
directories.

My PostgreSQL database is not huge, it is when packed about 50 MB. On
the file system it is 810 MB.

To select 2469 contacts as subset of 204048 contacts that belong in
certain group does not give (usually) feeling of any delay, it looks
instant for human.

My Org work is on meta-level so my truly important headings or subtree
names are in the database. Subtrees have their various properties,
like I can place any tags there inside, like TODO or designate type of
TODO. My work is intertwined with text and Org mode mostly, but I
could use any kind of mime type or any kind of Emacs mode. Some nodes
are on file system while some are in the database.

Nodes within subtree are hyperdocuments, they are all linkable and
could be on file system or not on file system.

Everything is together in one tree and it does not matter as access to
the nodes does not go over the tree necessary. There are 19197 nodes.
To find 76 that are tagged with TODO does not give me any slight or
visible delay, definitely not even 0.2 seconds. When I press enter it
is right there.

From the system I am using personally I am thinking that Org mode
could get its database connection so that headings and properties are
managed in the database fully while text could be managed in files. It
seems very possible.

The only thing that would be needed to add to Org in that case is some
heading tag that would uniquely designate where in the database that
heading is managed. It could be very lightly displayed on the screen
and would not be exported by default.

Something like

*** TODO Heading                                     :ID-123:

That would be all. All other meta data belonging to the heading could
be managed in the database. If heading is deleted it need not be
deleted in the database. Text belonging to heading could be managed in
the text file. Properties in the database. It can be simple database
such as GDBM if such is fast enough.

Meta data for the heading would or could be updated automatically from
time to time.

User could easily decide to show the properties in the Org file or not
to show. It does not matter much as long as :ID-123: tag is there.

All things like tags, properties, clock-in and out, schedule,
deadlines, custom_id and everything else as heading meta data could be
manageable in the database. It could be copied into new headings.
Creation of heading like this:

*** TitleRET

would automatically invoke creation of heading 124 in the database and it would appear as:

*** Title                                          :ID-124;

From there on user would be doing anything as usual in the Org mode
with the difference that properties would be displayed in the updated
manner and would not be really in the Org file. They would be
displayed on the fly. Any properties and plethora of other new
properties could be included.

System would recognize automatically by saving the Org file or by
opening it:

- If headings are in the right file, if file changed its place it
would be automatically updated in the database. 

- the heading ID would always remain unique no matter what, so users
linking to any heading would not need to worry of title remaining. The
unique ID that links to heading would basically link to the database
entry. Opening the link would ask database where the entry is located
and it would open up proper Org file at proper location without
parsing the Org file in usual manner. Org file would then remain
pretty much more text than it is now.

- all the parsing and searching and indexing would be automatically
solved and human readable SQL queries could be easily customized by
user. Suddenly there would be much less commotion in work. Org files
would look much more humane readable then they are now.

> 10k entries in a directory sounds inhumanely unergonomic.  I guess my
> biggest flat name directory might eventually reach that size?  In
> which case I could just split it in the middle of the alphabet, or
> similar solution.

Like by first letters, like

~/Maildir/a/d/a/adam@example.com

Such sorting of files would be automatic. You would need to invoke a
command that sorts files that way automatically and that may also
quickly access such files automatically.

I have comand that I often use, mkdatedir that makes me directory for
the current date.

If I wish to make a database note for the day, the command today-note would make sure there is:

- Year 2020 (formatted how I customize it)
  - November (also formatted by custom)
    - 2020-11-23
      And entry is automatically opened for the note.

The system helps that I locate quickly the note that relates to the
day. But I can put multiple notes under same date and I can also have
same titles for those multiple notes. This is because each note has
its unique ID.

I do not know how Org handles multiple same headings when linking to
it. It does not by default:

[[Heading][Heading]]

* Heading

  Text here
  
* Heading


  More text here. But if I wish to link here I need to do hac

To me and my thinkin that is not really logical. There shall be always
unique ID for each heading. My mind is not comforted by Org system in
that sense. And I should not be thinking of the unique ID neither I
should be writing those links like [[Something][Here]] as they should
be constructed automatically.

Myself I would like to come with cursor to second Heading and capture
the link to the heading. I would kill [[Heading][Heading]] into
special memory for those links. Then I could go to any other place in
the Org file and insert it there without thinking how link looks like
or constructing the link myself as it already exists in front of me.

Constructing links by hand is fine for those which are external.

Headings of Org files could be managed by the database in background.
Then all that distributed or sparse meta information (mess) disappears. 

What people are now trying to handle with Org files is management of a
database. Only that entries of the database are pretty much
disconnected from each other, vague, in unknown positions, then Org
algorhitms try to manage that all everything what is anyway built-in
in all SQL databases. Mess is growing over time.

> A 10k entry directory is getting into enterprise territory, and I'm
> sure enterprise has tech tricks that become worthwhile at that scale.

I will try with those options dir_index and dir_nlink to see if my
50000+ directory becomes somewhat faster. Direct access to the
subdirectory is always very fast. I almost never do ls there neither
enter any such directory manually. They store emails, so I just click
one key in mutt, that key extracts the current email address such as
person@example.com and opens up ~/Maildir/person@example.com, one
among 50000. It is accessed by wanting to see previous conversation
with the contact, not by knowing what is the directory name or email
address, computer does that. It is simple system I use for years and
it is blazing fast.

> There are scaling problems in every direction: Too many files per >
> directory, too large files, too much content per heading, too many >
> headings.

To list more than 200,000 contacts does take some time but access to
the list from database is so much faster than ls in the ~/Maildir with
more than 50000 entries or subdirectories. I can relate to that. And I
still think that file systems should manage any numbers of entries.

> There are scaling problems from too much deep tree nesting, namely too
> much fiddly ambiguous manual refiling.  Solution is flat "solid name"
> directories just below feasible 10 Bins.  Work fine.

I have tried your solution and could not find the mental concept to
relate to my thinking. And I do agree that such solution could help
other people.

For images I have some command like `sort-images.lisp' that just sorts
images by its embedded dates. Many times I sort even downloads per day.

* Texas Cyberthal <texas.cyberthal@gmail.com> [2020-11-23 12:51]:
> Hi Dr. Arne,
> 
> > The only part that hits performance limits is the agenda.
> 
> Well, IIRC your Org Textmind is much smaller than mine.
> 
> > My current guess is that the agenta is slow because it has to parse all my 7500 clock entries, and it has to check the Todo states of around 1200 headings.
> 
> Ouch.  I'd rather keep a "ramble log" so I can reconstruct an exactly
> honest time accounting, with discounts for partial attention, without
> worrying about fiddly clockin/outs.  At least when working from home.
> If clocking into a work site, that's different, because one can
> reasonably bill for the entire time, with minimal clock toggling.
> 

> > Did you check against filesystem limits? At 10k entries in a
> directory typical filesystems start becoming slow. That's the main
> reason I see for adding hierarchies.

From ext4 manual:

 dir_index
  Use hashed  b-trees to speed  up name lookups  in large
  directories.   This feature  is supported  by ext3  and
  ext4 file systems, and is ignored by ext2 file systems.

 dir_nlink
  This ext4 feature allows more than 65000 subdirectories
  per directory.

I think that file systems should be unlimited and fast in relation to
that. I have ~/Maildir with over 50000 subdirectories, direct access
is very easy and fast while listing takes some time.

If file system does not allow fast access it is time to replace it
with one that does allow it.

Now I wonder of HAMMER in DragonflyBSD is also slow with 50000
directories.

My PostgreSQL database is not huge, it is when packed about 50 MB. On
the file system it is 810 MB.

To select 2469 contacts as subset of 204048 contacts that belong in
certain group does not give (usually) feeling of any delay, it looks
instant for human.

My Org work is on meta-level so my truly important headings or subtree
names are in the database. Subtrees have their various properties,
like I can place any tags there inside, like TODO or designate type of
TODO. My work is intertwined with text and Org mode mostly, but I
could use any kind of mime type or any kind of Emacs mode. Some nodes
are on file system while some are in the database.

Nodes within subtree are hyperdocuments, they are all linkable and
could be on file system or not on file system.

Everything is together in one tree and it does not matter as access to
the nodes does not go over the tree necessary. There are 19197 nodes.
To find 76 that are tagged with TODO does not give me any slight or
visible delay, definitely not even 0.2 seconds. When I press enter it
is right there.

From the system I am using personally I am thinking that Org mode
could get its database connection so that headings and properties are
managed in the database fully while text could be managed in files. It
seems very possible.

The only thing that would be needed to add to Org in that case is some
heading tag that would uniquely designate where in the database that
heading is managed. It could be very lightly displayed on the screen
and would not be exported by default.

Something like

*** TODO Heading                                     :ID-123:

That would be all. All other meta data belonging to the heading could
be managed in the database. If heading is deleted it need not be
deleted in the database. Text belonging to heading could be managed in
the text file. Properties in the database. It can be simple database
such as GDBM if such is fast enough.

Meta data for the heading would or could be updated automatically from
time to time.

User could easily decide to show the properties in the Org file or not
to show. It does not matter much as long as :ID-123: tag is there.

All things like tags, properties, clock-in and out, schedule,
deadlines, custom_id and everything else as heading meta data could be
manageable in the database. It could be copied into new headings.
Creation of heading like this:

*** TitleRET

would automatically invoke creation of heading 124 in the database and it would appear as:

*** Title                                          :ID-124;

From there on user would be doing anything as usual in the Org mode
with the difference that properties would be displayed in the updated
manner and would not be really in the Org file. They would be
displayed on the fly. Any properties and plethora of other new
properties could be included.

System would recognize automatically by saving the Org file or by
opening it:

- If headings are in the right file, if file changed its place it
would be automatically updated in the database. 

- the heading ID would always remain unique no matter what, so users
linking to any heading would not need to worry of title remaining. The
unique ID that links to heading would basically link to the database
entry. Opening the link would ask database where the entry is located
and it would open up proper Org file at proper location without
parsing the Org file in usual manner. Org file would then remain
pretty much more text than it is now.

- all the parsing and searching and indexing would be automatically
solved and human readable SQL queries could be easily customized by
user. Suddenly there would be much less commotion in work. Org files
would look much more humane readable then they are now.

> 10k entries in a directory sounds inhumanely unergonomic.  I guess my
> biggest flat name directory might eventually reach that size?  In
> which case I could just split it in the middle of the alphabet, or
> similar solution.

Like by first letters, like

~/Maildir/a/d/a/adam@example.com

Such sorting of files would be automatic. You would need to invoke a
command that sorts files that way automatically and that may also
quickly access such files automatically.

I have comand that I often use, mkdatedir that makes me directory for
the current date.

If I wish to make a database note for the day, the command today-note would make sure there is:

- Year 2020 (formatted how I customize it)
  - November (also formatted by custom)
    - 2020-11-23
      And entry is automatically opened for the note.

The system helps that I locate quickly the note that relates to the
day. But I can put multiple notes under same date and I can also have
same titles for those multiple notes. This is because each note has
its unique ID.

I do not know how Org handles multiple same headings when linking to
it. It does not by default:

[[Heading][Heading]]

* Heading

  Text here
  
* Heading


  More text here. But if I wish to link here I need to do hack.

To me and my thinking that is not really logical. There shall be always
unique ID for each heading. My mind is not comforted by Org system in
that sense. And I should not be thinking of the unique ID neither I
should be writing those links like [[Something][Here]] as they should
be constructed automatically.

Myself I would like to come with cursor to second Heading and capture
the link to the heading. I would kill [[Heading][Heading]] into
special memory for those links. Then I could go to any other place in
the Org file and insert it there without thinking how link looks like
or constructing the link myself as it already exists in front of me.

Constructing links by hand is fine for those which are external.

Headings of Org files could be managed by the database in background.
Then all that distributed or sparse meta information (mess) disappears. 

What people are now trying to handle with Org files is management of a
database. Only that entries of the database are pretty much
disconnected from each other, vague, in unknown positions, then Org
algorhitms try to manage that all everything what is anyway built-in
in all SQL databases. Mess is growing over time.

> A 10k entry directory is getting into enterprise territory, and I'm
> sure enterprise has tech tricks that become worthwhile at that scale.

I will try with those options dir_index and dir_nlink to see if my
50000+ directory becomes somewhat faster. Direct access to the
subdirectory is always very fast. I almost never do ls there neither
enter any such directory manually. They store emails, so I just click
one key in mutt, that key extracts the current email address such as
person@example.com and opens up ~/Maildir/person@example.com, one
among 50000. It is accessed by wanting to see previous conversation
with the contact, not by knowing what is the directory name or email
address, computer does that. It is simple system I use for years and
it is blazing fast.

> There are scaling problems in every direction: Too many files per >
> directory, too large files, too much content per heading, too many >
> headings.

To list more than 200,000 contacts does take some time but access to
the list from database is so much faster than ls in the ~/Maildir with
more than 50000 entries or subdirectories. I can relate to that. And I
still think that file systems should manage any numbers of entries.

> There are scaling problems from too much deep tree nesting, namely too
> much fiddly ambiguous manual refiling.  Solution is flat "solid name"
> directories just below feasible 10 Bins.  Work fine.

I have tried your solution and could not find the mental concept to
relate to my thinking. And I do agree that such solution could help
other people.

For images I have some command like `sort-images.lisp' that just sorts
images by its embedded dates. Many times I sort even downloads per day.

Memacs tries to solve about same problem.

Memacs
https://github.com/novoid/Memacs

That hyperlink I have selected among other 20000 hyperlinks. I could
as well send the notes to you or annotation related to the
hyperlink. I have not written the hyperlink myself, all I did is that
I have opened HyperScope, invoked completion and on the link on screen
I pressed W, it copied itself to this email. It was blazing fast as I
have accessed it by thinking Memex. Not Memacs, but Memex. Memacs was
just next to it. By thinking would still mean that I had to enter some
words that I think of. Memory is involved in that process of thinking
and accessing.

You mentioned humans know many words. If we observe the process of
knowing words, how do we access them? This time really by
thinking. But we access them how I heard of it, mostly by association
or by direct access. We see the flower and word is just there.

Do we think of a tree of knowledge first? I do not think so. And there
are memory systems that DO think of plethora of various things and
increase human memory capabilities. That is called
mnemonics. Mnemonics is based mostly on associations. It becomes
possible to remember pack of mixed 52 cards within 20 minutes and to
reliably know at which position is which card located and to replicate
the full series of cards. Mnemonics methods help human to do such
feats. Everybody can do it.

Compare now the human mind system:

- of direct access by direct association, something like I think of
  Memex but I know there is something similar in Emacs, I write Memex
  and I get Memacs, then I give reference to you.

- and there is also the system of thinking that I can locate in my
  mind a reference to Memacs even by its number or ID because that
  could be my mnemonics how I think about

How human think -- is nowhere defined and is vague. Human thinks how
they think and there may be as many versions as humans.

Computers should not be delivered any more with one built-in paradigm
only such as file system. There shall be at least several:

- file system

- meta databased approach, that involves little but more curation than
  just making a file name.

- subject or tag based approach

- Dewey decimal approach or other similar

- 10 Bins, etc.

Then user could decide to use this or that approach. Having file
managers for decades is really boring. It does not advance computing.
To say that we have hierarchical file systems by default and nothing
else shows how much we are under-developed. 

Doug Engelbart has already envisioned how files could be stored,
accessed, hyperlinked, referenced and we do not use it in that sense
today after how many years? Maybe 40 years. Computer makers and OS
makers do not really help us, there are visionaries but we do not get
file systems that helps us to access files by association or thinking
so we have to upgrade our tools for those tasks that should be built
in.

Org as a concept was already invented by Doug Engelbart before decades
and it still does not have features that I would like it to have. For
example finely grained unique ID numbers that can also relate to
paragraphs or set of paragraphs, unique or static sorting of files
repository, wide group collaboration and sharing and other concepts.
Hyperlinking already back than was sophisticated.

Highlights of the 1968 "Mother of All Demos"
https://www.dougengelbart.org/content/view/276/000/

Re: One vs many directories

[Ihor Radchenko]

Dear Jean Louis,

Your description of the database reminds me how org-roam handles the
files - it also uses an external database for linking and allows quick
incremental search that does not really depend on where the
files/headings are stored.

However, what you are talking about is against org-mode philosophy, as I
know it. Currently, the dev's stance is that org files must be
self-sufficient. Org-mode should not depend on external database to
manage the org files and operations with them. Everything must be plain
text! Moreover, some devs are even reluctant to hide metadata (like
unique ID implemented in org-id module) from users (which is possible
and also partially implemented).

Best,
Ihor


Jean Louis <bugs@gnu.support> writes:

> * Texas Cyberthal <texas.cyberthal@gmail.com> [2020-11-23 12:51]:
>> Hi Dr. Arne,
>> 
>> > The only part that hits performance limits is the agenda.
>> 
>> Well, IIRC your Org Textmind is much smaller than mine.
>> 
>> > My current guess is that the agenta is slow because it has to parse all my 7500 clock entries, and it has to check the Todo states of around 1200 headings.
>> 
>> Ouch.  I'd rather keep a "ramble log" so I can reconstruct an exactly
>> honest time accounting, with discounts for partial attention, without
>> worrying about fiddly clockin/outs.  At least when working from home.
>> If clocking into a work site, that's different, because one can
>> reasonably bill for the entire time, with minimal clock toggling.
>> 
>
>> > Did you check against filesystem limits? At 10k entries in a
>> directory typical filesystems start becoming slow. That's the main
>> reason I see for adding hierarchies.
>
> From ext4 manual:
>
>  dir_index
>   Use hashed  b-trees to speed  up name lookups  in large
>   directories.   This feature  is supported  by ext3  and
>   ext4 file systems, and is ignored by ext2 file systems.
>
>  dir_nlink
>   This ext4 feature allows more than 65000 subdirectories
>   per directory.
>
> I think that file systems should be unlimited and fast in relation to
> that. I have ~/Maildir with over 50000 subdirectories, direct access
> is very easy and fast while listing takes some time.
>
> If file system does not allow fast access it is time to replace it
> with one that does allow it.
>
> Now I wonder of HAMMER in DragonflyBSD is also slow with 50000
> directories.
>
> My PostgreSQL database is not huge, it is when packed about 50 MB. On
> the file system it is 810 MB.
>
> To select 2469 contacts as subset of 204048 contacts that belong in
> certain group does not give (usually) feeling of any delay, it looks
> instant for human.
>
> My Org work is on meta-level so my truly important headings or subtree
> names are in the database. Subtrees have their various properties,
> like I can place any tags there inside, like TODO or designate type of
> TODO. My work is intertwined with text and Org mode mostly, but I
> could use any kind of mime type or any kind of Emacs mode. Some nodes
> are on file system while some are in the database.
>
> Nodes within subtree are hyperdocuments, they are all linkable and
> could be on file system or not on file system.
>
> Everything is together in one tree and it does not matter as access to
> the nodes does not go over the tree necessary. There are 19197 nodes.
> To find 76 that are tagged with TODO does not give me any slight or
> visible delay, definitely not even 0.2 seconds. When I press enter it
> is right there.
>
> From the system I am using personally I am thinking that Org mode
> could get its database connection so that headings and properties are
> managed in the database fully while text could be managed in files. It
> seems very possible.
>
> The only thing that would be needed to add to Org in that case is some
> heading tag that would uniquely designate where in the database that
> heading is managed. It could be very lightly displayed on the screen
> and would not be exported by default.
>
> Something like
>
> *** TODO Heading                                     :ID-123:
>
> That would be all. All other meta data belonging to the heading could
> be managed in the database. If heading is deleted it need not be
> deleted in the database. Text belonging to heading could be managed in
> the text file. Properties in the database. It can be simple database
> such as GDBM if such is fast enough.
>
> Meta data for the heading would or could be updated automatically from
> time to time.
>
> User could easily decide to show the properties in the Org file or not
> to show. It does not matter much as long as :ID-123: tag is there.
>
> All things like tags, properties, clock-in and out, schedule,
> deadlines, custom_id and everything else as heading meta data could be
> manageable in the database. It could be copied into new headings.
> Creation of heading like this:
>
> *** TitleRET
>
> would automatically invoke creation of heading 124 in the database and it would appear as:
>
> *** Title                                          :ID-124;
>
> From there on user would be doing anything as usual in the Org mode
> with the difference that properties would be displayed in the updated
> manner and would not be really in the Org file. They would be
> displayed on the fly. Any properties and plethora of other new
> properties could be included.
>
> System would recognize automatically by saving the Org file or by
> opening it:
>
> - If headings are in the right file, if file changed its place it
> would be automatically updated in the database. 
>
> - the heading ID would always remain unique no matter what, so users
> linking to any heading would not need to worry of title remaining. The
> unique ID that links to heading would basically link to the database
> entry. Opening the link would ask database where the entry is located
> and it would open up proper Org file at proper location without
> parsing the Org file in usual manner. Org file would then remain
> pretty much more text than it is now.
>
> - all the parsing and searching and indexing would be automatically
> solved and human readable SQL queries could be easily customized by
> user. Suddenly there would be much less commotion in work. Org files
> would look much more humane readable then they are now.
>
>> 10k entries in a directory sounds inhumanely unergonomic.  I guess my
>> biggest flat name directory might eventually reach that size?  In
>> which case I could just split it in the middle of the alphabet, or
>> similar solution.
>
> Like by first letters, like
>
> ~/Maildir/a/d/a/adam@example.com
>
> Such sorting of files would be automatic. You would need to invoke a
> command that sorts files that way automatically and that may also
> quickly access such files automatically.
>
> I have comand that I often use, mkdatedir that makes me directory for
> the current date.
>
> If I wish to make a database note for the day, the command today-note would make sure there is:
>
> - Year 2020 (formatted how I customize it)
>   - November (also formatted by custom)
>     - 2020-11-23
>       And entry is automatically opened for the note.
>
> The system helps that I locate quickly the note that relates to the
> day. But I can put multiple notes under same date and I can also have
> same titles for those multiple notes. This is because each note has
> its unique ID.
>
> I do not know how Org handles multiple same headings when linking to
> it. It does not by default:
>
> [[Heading][Heading]]
>
> * Heading
>
>   Text here
>   
> * Heading
>
>
>   More text here. But if I wish to link here I need to do hac
>
> To me and my thinkin that is not really logical. There shall be always
> unique ID for each heading. My mind is not comforted by Org system in
> that sense. And I should not be thinking of the unique ID neither I
> should be writing those links like [[Something][Here]] as they should
> be constructed automatically.
>
> Myself I would like to come with cursor to second Heading and capture
> the link to the heading. I would kill [[Heading][Heading]] into
> special memory for those links. Then I could go to any other place in
> the Org file and insert it there without thinking how link looks like
> or constructing the link myself as it already exists in front of me.
>
> Constructing links by hand is fine for those which are external.
>
> Headings of Org files could be managed by the database in background.
> Then all that distributed or sparse meta information (mess) disappears. 
>
> What people are now trying to handle with Org files is management of a
> database. Only that entries of the database are pretty much
> disconnected from each other, vague, in unknown positions, then Org
> algorhitms try to manage that all everything what is anyway built-in
> in all SQL databases. Mess is growing over time.
>
>> A 10k entry directory is getting into enterprise territory, and I'm
>> sure enterprise has tech tricks that become worthwhile at that scale.
>
> I will try with those options dir_index and dir_nlink to see if my
> 50000+ directory becomes somewhat faster. Direct access to the
> subdirectory is always very fast. I almost never do ls there neither
> enter any such directory manually. They store emails, so I just click
> one key in mutt, that key extracts the current email address such as
> person@example.com and opens up ~/Maildir/person@example.com, one
> among 50000. It is accessed by wanting to see previous conversation
> with the contact, not by knowing what is the directory name or email
> address, computer does that. It is simple system I use for years and
> it is blazing fast.
>
>> There are scaling problems in every direction: Too many files per >
>> directory, too large files, too much content per heading, too many >
>> headings.
>
> To list more than 200,000 contacts does take some time but access to
> the list from database is so much faster than ls in the ~/Maildir with
> more than 50000 entries or subdirectories. I can relate to that. And I
> still think that file systems should manage any numbers of entries.
>
>> There are scaling problems from too much deep tree nesting, namely too
>> much fiddly ambiguous manual refiling.  Solution is flat "solid name"
>> directories just below feasible 10 Bins.  Work fine.
>
> I have tried your solution and could not find the mental concept to
> relate to my thinking. And I do agree that such solution could help
> other people.
>
> For images I have some command like `sort-images.lisp' that just sorts
> images by its embedded dates. Many times I sort even downloads per day.

Is Org really so simple?

[Jean Louis]

* Ihor Radchenko <yantar92@gmail.com> [2020-11-23 17:18]:
  :PROPERTIES:
  :CREATED:  [2020-11-23 Mon 18:42]
  :ID:       edebb3e7-e755-4ecc-a5e8-a3353a3f5fd0
  :END:
> Dear Jean Louis,
> 
> Your description of the database reminds me how org-roam handles the
> files - it also uses an external database for linking and allows quick
> incremental search that does not really depend on where the
> files/headings are stored.

Sounds good, I can see there is graph database used.

> However, what you are talking about is against org-mode philosophy,
> as I know it.

Only philosophy I know is that it is plain text. Is there any official
philosophy? I have no idea, at least manual does not give me
references. I cannot find "philosophy", send me references.

It says "to keep simple things simple". But Org is far far from being
simple any more. It offers good principles, paradigms and people built
many enhancements upon those. Speedily it becomes way much more than
simple.

Headings do not look any more as headings, it looks like pieces of
code to a person that is new. Properties, tags, clocks, schedule,
deadline, all that tries to store itself under specific heading. There
is easily too much of it and things are not simple any more.

> Currently, the dev's stance is that org files must be
> self-sufficient.

There is no compact principle there practically. Anything is
possible. That Org files are not practically self-sufficient shows the
fact that there are 129 Emacs packages in one Org distribution. 

> Org-mode should not depend on external database to manage the org
> files and operations with them. Everything must be plain text!

Question is what is meant by database, it can be anything. One can
save LISP data. Recent files, desktop, eww bookmarks, init.el or
.emacs file are also all similar databases, there is the underused
EIEIO with persistent stuff that represent built-in database
functionality.

That Org files are not self-sufficient shows the demand that there is
almost no Org user who does not have add-ons, packages, modifications,
configurations.

Would it be really self-sufficient there would be no development going
on, right?

Babel executions clearly show that Org is not self sufficient and
depends on number of external software.

I don't mind of philosophy, in fact I would like that philosophy is
really that what it wanted to be, but that time is over.

I am just pointing out that it is by many means not self sufficient.

Is by default LaTeX export enabled? I think it is. How big is the
LaTeX package? It is huge, and Org depends on it for export.

Thus Org is far far from being self-sufficient.

Almost every system has GDBM database, if Emacs would have bindings to
GDBM, there would be so much less of development in general for many
various packages and Emacs would be expanding faster and easier.

In fact I think that author and initial developers could not predict
at the time where the Org goes and that speaking of self-sufficient
and "plain text" only is history.

Before I found out about Org I was using back in time `hnb' in console
to track various planning tasks. It works nice and simple. That is
really self sufficient. Org definitely not.

HNB - Hierarchical Notebook
http://hnb.sourceforge.net/

In the mean time I have created database where I can store TODO,
Notes, Calls, SMS, People, Invoices, Groups, Mailing Lists and so on,
and made my own shell and Perl interfaces to it. And I used to manage
it through: GeDaFe - PostgreSQL Generic Database Interface
http://gedafe.github.io/doc/gedafe-sql.en.html and this was and is
hierarchical or better graph knowledge management and relational
database.

Creating simple table in the database automatically helped me to
manage that table. It is trivial to create NOTES table with schedule
date, clock in, TODO or other conditions and tags. The interface is
just there and automatic to whatever table I create the interface is
there to add/modify/delete/search/refine records. That is what I would
say "simple" and keeping things simple and indefinitely extensible
without modification of software. The fundamental GeDaFe principle I
would like to try to implement in Emacs. And same database I use for
web interface I am using also within Emacs.

GeDaFe principle is following:

- define your data (like handling notes, TODO, or executing scripts within notes)

- work and forget about any underlying functions,
  add/create/delete/modify/index or search, make reports with simple exports

- expand with more definitions of data when necessary (like add
  various properties, or other data tables, like contacts, invoices,
  etc.) and repeat the process.

Org also shows that it does not hold "Notes" only, it holds more than
that, I have written average book size technical documents with
it. Only just one part of the document is printed from one single node
that belongs to single project among many. People use such documents
on the ground. My use case is not for simple notes.

A node in a tree can be anything, and Org enhancements develop by that
principle. For example there is org-contacts where nodes are meant to
be "People". Such development is so much hard coded.

Simpler would be to define the type of nodes and work by their
types. One could need just one type table and one node table and that
is about all.

The type table could say:

- this node is heading
- or, this node is text under heading
- or, this node is paragraph among many others
- or, this node is hyperlink to WWW URI
- or, this node is hyperlink to file
- this node is movie to play
- this type is PDF to be opened on page 3
- this one is really note
- this one is note but with action assigned like TODO, etc.

Nodes could have its properties defined like for anything. Nodes can
reference its parent nodes. Node can be parent to any heading.

Once such 2 tables are defined magic starts to take place, it becomes
its subtree with all kinds of node types including Babel execution and
similar. Meta data is meta, it can be updated but need not be
visible. Computer is handling meta data.

Node can be a page in a subtree that becomes a website.

Node can be object for person or company, or just anything.

I am currently using my nodes to quickly research various subjects by
using that type of dynamic knowledge repository.

Org file is dynamic knowledge repository.

About Dynamic Knowledge Repositories (DKR)
https://www.dougengelbart.org/content/view/190/163/

Then around 2015 I have discovered Cherrytree and have made bunch of
notes with it, and that is self-sufficient one program that keeps
simple things simple as it is much more simpler than Org. It offers a
visual interface to all features related to the knowledge management

Cherrytree - hierarchical note taking application with rich text and syntax highlighting
https://www.giuspen.com/cherrytree/

TAGS in Cherrytree are automatic if I remember well, TAG is simply
name of the node. If I call node TODO, then nodes under are simply
TODO items. 

Later I discovered there is something similar in Emacs so I started
with Org and use it mostly for instruction writing and project
management. And I find many options over kill for me. On the other
hand my usage of Org would be overkill for somebody, so it depends of
viewpoints.

In general I was always using headings and subheadings, trees, lists, notes by using text.

Somewhere 2004 I started using Markdown one among first as I found it
simpler than ASCIIDOC and M4, but not as perfect.

2020 and 2021 I like to raise the level of dynamic knowledge
journaling to the meta level where I think less about underlying
functions in software.

That experience also tells that Org is definitely not simple.

Among hnb, GeDaFe database approach, Cherrytree and Org mode, for
"keeping things simple" like note taking and TODO items, project
management, Cherrytree was the best for me.

Among all those for keeping complex things simple the database
approach is the best. Of course that I use database within Emacs and
it is not visible to user what it is. Database allows simultaneous
operation by several people on distance and that is the groupware
feature as envisioned by Doug Engelbart.

For document writing and nice formatting with LaTeX export, Org mode
is the best personally.

For notes, database based notes are best as only so I have relations
between the note and other objects. With 200,000 contacts in the
database which I can quickly access and assign notes to them, how
would that work with Org? Hardly. Notes are related to people, to
projects, plans, opportunities, research subjects, companies and so
on. There is no simple way in Org mode to relate one note to multiple
other related subjects.

And people try to do exactly that, people are developing Org in the
manner to relate objects in Org file to anything else. And they do
hard work as they do it manually. Relational database speeds up and
does not tell user to manually hyperlink various relations.

I have sent thousands of tasks to people by using this function
below. And I had to define for each Org file who are "assignees" for
that Org file. And I have like 50 assignees, so I need to copy and
paste their nick names or identifiers into the Org file. There it
comes the attribute of being "self-sufficient", it becomes
self-sufficient because I had to define all assignees into that
specific file, but I find it tedious and not useful.

(defun rcd/org-send-assigned-task ()
  "Sends assigned task to designated individual as Org"
  (interactive)
  (let* ((member-data (rcd-org-extract-assigned-member-email-data))
	 (id (if member-data (first member-data) nil))
	 (signature (if (equal (type-of (symbol-value (fifth member-data))) 'cons)
	 		(third (symbol-value (fifth member-data))) ""))
	 (file (rcd-org-subtree-to-file signature))
	 (subject (rcd/org-find-headline))
	 (esubject (escape-% subject))
	 (ask (unless id (y-or-n-p "No assignment found. Do you want to send it by email?")))
	 (name (if id (third member-data)))
	 ;; (name (if ask (read-from-minibuffer "Name:") name))
	 (voice (format "The task '%s' is being sent to '%s'" subject name))
	 (email (if id (if (equal (type-of (fourth member-data)) 'cons)
			   (car (fourth member-data))
			 (fourth member-data))))
	 (email (if ask (cf-search-email (read-from-minibuffer "Search for email: ")) email))
	 (really (y-or-n-p (format "Do you really want to send it to: %s?" (if ask email name)))))
    (if (and really (or id ask))
      (if (string-match "@" email)
	(progn
	  ;; (message (escape-% subject))
	  (speak voice)
	  (mutt-send-file name email esubject file))
	(message "No email specified"))
      (message "Aborted sending."))))

> Moreover, some devs are even reluctant to hide metadata (like unique
> ID implemented in org-id module) from users (which is possible and
> also partially implemented).

I hope that I have demonstrated that one who develops could review
several various paradigms and learn more. I am fine with any decision
of design for Org mode as I use it as it is and I have for me other
ways of managing data. I am not sure how much those features have been
discussed to say that hiding meta data is good or not good. It is
better to discuss and find what is more useful.

What I can see is that people complain for speed and they build
extensions that help with it. Extensions are external while built-in
Org does not keep up with the dynamic how people expect it to be.

For example I would expect Org to be very simple, very very simple, I
would rewind it back to its roots. Somebody else would like
complexities. Currently I can see that Org is not made for
complexities. It is better to unwind the development and make Org in
core very basic and speedy and let people enhance it with external
packages. In general it is better to remain simple. Even that may not
be possible any more.

I see hard work by many people who try to enhance Org as hierarchical
knowledge of data because people want to have feature X, but group of
those enhancements in reality belong to relational databases and not
to text files. Developers wish to accommodate various people and yet
by doing so they develop it into complex software. (129 .el packages
for one Emacs mode!?)

Among those one shall read the Doug Engelbart's paradigms, then
especially if one is developer of system of data management that many
people use, one shall explore other paradigms, including various
approaches to data handling. And overall one shall keep it simple as
it is main fundament of Org to be simple, while practical fact remains
that it is not anymore simple, not at all. 

I remember back in time with BASIC programming language that it had
also something like a built-in database that one could put on the
bottom of the file. Then there is also Perl's __DATA__ that is placed
straight into the file and one could also place images and other
stuff. Maybe the meta data could be kept this way in just one heading
named Meta data, and this heading would not be exported, it could be
hidden from the user and it could contain the database necessary for
single Org file. By pressing key to show properties one could see
properties or simply make them hidden. By making a copy of subtree the
metadata would also copy as usual. By exporting, one could make Org
without meta data, and I like using this information as I like sending
Org headings without personal meta data to third parties.

Re: Is Org really so simple?

[Tom Gillespie]

I have read many perspectives like this of late on this mailing list.
In summary I think that Org is such an incredibly flexible and
powerful tool that many users have not the faintest idea what other
users are doing with it (for example I am completely mystified by the
level of activity in the one vs many files thread and its counterpart
on the orgmode subreddit). Despite this, in a sense, Org is just as
simple as it has always been which is why people build on top of it,
and thus there isn't really any way to "go back" to a simpler time --
such a time is fictional, it has never existed. I can say for myself
that it is not Org that has changed, it is how I use it. I used to use
it in simple ways, and still can if I want, but now I use it as a
substrate for self describing (sometimes self-modifying) interactive
programs -- as complex as you can get.

For some use cases there are performance issues and for others
workflow issues. The performance issues are likely to be resolved via
more developer time. The workflow issues have to be resolved by
writing custom elisp code, in many times that elisp gets packaged as
org extensions. Thus we arrive back at your original complaint, which
is that Org files are not sufficiently self-contained. In order to
make them self-contained you currently have to copy and paste a bunch
of boilerplate into files (thus the workflow issues). For some things
the boilerplate is seemingly unavoidable, even using #+setupfile:
doesn't work if you are trying to solve a problem in a certain way in
Org. Other times, you could solve the same problem without any
boilerplate using a slightly different approach, but not always.

I have been working on an extension for Org (orgstrap
https://github.com/tgbugs/orgstrap) with the goal of making Org files
self-describing, if not fully self-contained (There is a distinction
between the two, but only for certain failure modes. Also, why force
__DATA__ to be embedded with the file when everything has to be
dereferenced at some point anyway? You could embed it later if it fits
your use case, or you could embed the org file in the data!). I came
at the problem from the perspective of reproducible research, but it
seems like there are many use cases for being able to move information
implicit in the environment into an Org file explicitly. If you want a
database to handle relational data, great, describe what you need to
access it in the org file that will act as the interface to that
relational data, that way if the database access is down you won't get
cryptic errors, but a big warning that says "hey! a service required
to use functionality in this file correctly is missing!" For the
described frustration with needing to track users, give the org-file
an embedded id and use that to access and update the assignees
associated with a file. There is no need to choose Org or something
else, you can have your comfortable org-mode workflow, and your
data-appropriate store.

Re: Is Org really so simple?

[Jean Louis]

* Tom Gillespie <tgbugs@gmail.com> [2020-11-23 23:41]:
> I have read many perspectives like this of late on this mailing list.
> In summary I think that Org is such an incredibly flexible and
> powerful tool that many users have not the faintest idea what other
> users are doing with it (for example I am completely mystified by the
> level of activity in the one vs many files thread and its counterpart
> on the orgmode subreddit).

I have many files but I do not hyperlink them as they are in the other
sense self-contained. This is due to my individual use case. I simply
do not know why should I hyperlink from one file to other as subjects
of the files are so much separate.

My way of handling things is to designate a project with TODOs, but I
do not have TODOs in many various files so those agenda adding
functions are rather disturbing me then helping as those are few
seconds of delay for nothing. My TODO things are in separate files and
they are many times not really "My TODO". They are rather TODO for
people to which I assign tasks, and send such. Those people do not
need to write Org files, they get it by email and execute in the real
world, then they report back.

When searching for tasks I do not use agenda, I use people from where
I quickly enter the Org file related to that person without knowing
where it is located, there are many Org files like that.

That is how I do not encounter problems others do encounter.

One bulk of 173 Org files in one single directory I do not search with
Agenda rather with grep as those are simply not TODO/Action related.

There are just 2-3 files that have TODO/Action related stuff and they
are on key bindings.

I am in transition to stop with Org tracking my TODO actions and just
use it for the written instructions, prose and project management
which is then printed on the paper and assigned to people. While my
TODO notes I will simply keep in the database as it gives me better
access and tracking.

Just recently I have implemented dumb version control or backup for
database entries so that entry is first saved before editing for later
comparison or revisions. If I am to use RCS or other revision control
system on user's Org files, this would mean I would need to invoke the
version control in hundreds of directories and use all the
keybindings, etc. With the program backed version control when editing
Org file from database, it is automatically saved in the `vc' table
without me thinking anything about it.

If there is any version control, users should not think about it. It
should be a built-in feature. 

> Despite this, in a sense, Org is just as simple as it has always
> been

On that I am not convinced at all. 129 Emacs .el files are in the Org
distribution here and that I do not call simple. Maybe outline-mode is
simple on which org is built upon. 

> which is why people build on top of it, and thus there isn't really
> any way to "go back" to a simpler time -- such a time is fictional,
> it has never existed.

Right.

> I can say for myself that it is not Org that has changed, it is how
> I use it. I used to use it in simple ways, and still can if I want,
> but now I use it as a substrate for self describing (sometimes
> self-modifying) interactive programs -- as complex as you can get.

I will look into that package to see if it has good ideas for
HyperScope dynamic knowledge repository that I am working on,
something like meta Org.

> For some use cases there are performance issues and for others
> workflow issues.

I have no workflow issues and performance only when Org updates its
stuff which I do not need. I would like to see some more automated
link construction functions and recognitions of other modes. For
example when browsing gopher:// or gemini:// pages with M-x elpher
that I can quickly construct hyperlinks and insert somewhere. Org has
such functions to read email and obtain hyperlink, to use eww and
obtain hyperlink and similar. It just needs more and
more. Hyperlinking is what I need but I need it in more textual and
peculiar non-common way.

I would rather like to have simple text files formatted any how so
that file can be readable and sent to other people while meta
information and hyperlinks can be defined in separate file or other
pieces of information. GNU Hyperbole offers similar system where
buttons and hyperlink information is defined in separate files. I can
include GNU Hyperbole hyperlinks in any file and they will not look so
complex as Org files.

This link looks complicated if not in Org mode:

[[https://www.example.com/something/more/here][Fetch software]]

This link can be in any mode and looks simpler:

<(Fetch software)> 

but in that case hyperlinks are defined elsewhere, not in same file.

> Thus we arrive back at your original complaint, which is that Org
> files are not sufficiently self-contained.

No complaint, just my viewpoint. I do not mind of principles,
philosophy, etc. I do not need Org files to be sufficiently
self-contained.

I have some preferences but for Org I do not mind and accept it as
such. As said, I would prefer having text files which can be upgraded
with meta file to show all the hyperlinks. That is what I am using now
in HyperScope.

> In order to make them self-contained you currently have to copy and
> paste a bunch of boilerplate into files (thus the workflow
> issues).

Yes and no. I think that what you explained is not something to think
of. That is why when I create a person, like Karl, I can locate person
in the database and press F4 and Org file is created specifically for
that person with all details inside, like title, author, assignmets,
few basic headings such as Tasks and Transactions including the table
of transactions.

If I am thinking of groups of people, there is artist group and staff
member group, so there could be templates for artist group and staff
members, and by simply clicking on one single key the Org file is
created on the fly for that specific person without me knowing where
the file is located or putting any hand written templates. It is just
there, already saved and opened and I do not even need to save
it. Click and go.

> I have been working on an extension for Org (orgstrap
> https://github.com/tgbugs/orgstrap ) with the goal of making Org files
> self-describing, if not fully self-contained (There is a distinction
> between the two, but only for certain failure modes. Also, why force
> __DATA__ to be embedded with the file when everything has to be
> dereferenced at some point anyway?

I have just given idea that meta data like properties and anything
else could be defined in separate section of same file which in turn
could export itself to full Org or something else.

Separate section of Org file or separate meta data file could provide
hyperlinks for specific keywords without those being hyperlinsk in the
original file. Switching between Org type view and meta Org type view
could be easy. The idea is there just for thinking, I will definitely
not work on such as I work different

> You could embed it later if it fits your use case, or you could
> embed the org file in the data!).

In that sense I have database columns that hold Org data, not files. I
edit such just as usual, it is just not on the file system and it does
not need to be Org file, it can be just any type of a file that Emacs
support and any type of a mode. That way I am getting meta level
hierarchy of hyperdocuments that can be just anything.

> If you want a database to handle relational data, great, describe
> what you need to access it in the org file that will act as the
> interface to that relational data, that way if the database access
> is down you won't get cryptic errors, but a big warning that says
> "hey! a service required to use functionality in this file correctly
> is missing!"

I would not like creating too many things by hand, that is why
computer has to take those tasks from me. So I am working other way
around like adding records to the database and then viewing stuff with
the Org. Database is exported to Org buffer, not file, but could be
saved as file and then all relational elements and hyperlinks can be
already there automatically. If person Karl received email ABC, that
ABC can be hyperlink to the email, I can see all SMS, previous email
conversation and so on and that works by clicking on the automatically
generated hyperlinks. It would be pain in the ass that information
that is already available to be collected by computer I have to
collect with my efforts. Thus any pieces of information that already
exists should be converted on the fly to hyperlinks to hyperdocuments
without user having to construct it by hand.

Hand work is for those files where one wish to create references for
the first time. But if any referencable objects already exists and are
related to any other objects than Org files can be created on the fly
to inspect such relations.

Re: Is Org really so simple?

[Ihor Radchenko]

> Only philosophy I know is that it is plain text. Is there any official
> philosophy? I have no idea, at least manual does not give me
> references. I cannot find "philosophy", send me references.

You are right. There is no official "philosophy" in org. In my reply I
tried to follow the topic starter's view:

 Texas Cyberthal <texas.cyberthal@gmail.com>:
> By philosophy, I mean the dev consensus on the correct way to do
> things, and coded configuration and usability biases.

According to my experience with org-mode development (I am not talking
about third-party packages here), it is discouraged to change org-mode
towards hiding metadata in org files or store *unique* data (that cannot
be derived from the contents of the org files) related to org-mode
externally (not in org files). It is not official statement, but rather
my impression so far.

> Question is what is meant by database, it can be anything. One can
> save LISP data. Recent files, desktop, eww bookmarks, init.el or
> .emacs file are also all similar databases, there is the underused
> EIEIO with persistent stuff that represent built-in database
> functionality.

Org-mode does not limit user customisations. It does not limit
third-party packages either. Users are free to use any other tools,
store their data anywhere other than org files (why would org force
users to do otherwise?). What I referred to earlier is just the core
org-mode development.

> And people try to do exactly that, people are developing Org in the
> manner to relate objects in Org file to anything else. And they do
> hard work as they do it manually. Relational database speeds up and
> does not tell user to manually hyperlink various relations.

Could you provide some more examples? I do not see how relational
database is different from creating hyperlinks in org. Either way, the
user needs to file an object/headline somewhere into org file, which is
inherently manual.

> I see hard work by many people who try to enhance Org as hierarchical
> knowledge of data because people want to have feature X, but group of
> those enhancements in reality belong to relational databases and not
> to text files.

It is an interesting point. I would be happy if some existing tools
could be reused instead of re-inventing the wheel for org. Do you have
concrete examples where it can be useful? If you have, I encourage you
to bring up a feature request to discuss this possibility with org-mode
devs.

Best,
Ihor

Jean Louis <bugs@gnu.support> writes:

> * Ihor Radchenko <yantar92@gmail.com> [2020-11-23 17:18]:
>   :PROPERTIES:
>   :CREATED:  [2020-11-23 Mon 18:42]
>   :ID:       edebb3e7-e755-4ecc-a5e8-a3353a3f5fd0
>   :END:
>> Dear Jean Louis,
>> 
>> Your description of the database reminds me how org-roam handles the
>> files - it also uses an external database for linking and allows quick
>> incremental search that does not really depend on where the
>> files/headings are stored.
>
> Sounds good, I can see there is graph database used.
>
>> However, what you are talking about is against org-mode philosophy,
>> as I know it.
>
> Only philosophy I know is that it is plain text. Is there any official
> philosophy? I have no idea, at least manual does not give me
> references. I cannot find "philosophy", send me references.
>
> It says "to keep simple things simple". But Org is far far from being
> simple any more. It offers good principles, paradigms and people built
> many enhancements upon those. Speedily it becomes way much more than
> simple.
>
> Headings do not look any more as headings, it looks like pieces of
> code to a person that is new. Properties, tags, clocks, schedule,
> deadline, all that tries to store itself under specific heading. There
> is easily too much of it and things are not simple any more.
>
>> Currently, the dev's stance is that org files must be
>> self-sufficient.
>
> There is no compact principle there practically. Anything is
> possible. That Org files are not practically self-sufficient shows the
> fact that there are 129 Emacs packages in one Org distribution. 
>
>> Org-mode should not depend on external database to manage the org
>> files and operations with them. Everything must be plain text!
>
> Question is what is meant by database, it can be anything. One can
> save LISP data. Recent files, desktop, eww bookmarks, init.el or
> .emacs file are also all similar databases, there is the underused
> EIEIO with persistent stuff that represent built-in database
> functionality.
>
> That Org files are not self-sufficient shows the demand that there is
> almost no Org user who does not have add-ons, packages, modifications,
> configurations.
>
> Would it be really self-sufficient there would be no development going
> on, right?
>
> Babel executions clearly show that Org is not self sufficient and
> depends on number of external software.
>
> I don't mind of philosophy, in fact I would like that philosophy is
> really that what it wanted to be, but that time is over.
>
> I am just pointing out that it is by many means not self sufficient.
>
> Is by default LaTeX export enabled? I think it is. How big is the
> LaTeX package? It is huge, and Org depends on it for export.
>
> Thus Org is far far from being self-sufficient.
>
> Almost every system has GDBM database, if Emacs would have bindings to
> GDBM, there would be so much less of development in general for many
> various packages and Emacs would be expanding faster and easier.
>
> In fact I think that author and initial developers could not predict
> at the time where the Org goes and that speaking of self-sufficient
> and "plain text" only is history.
>
> Before I found out about Org I was using back in time `hnb' in console
> to track various planning tasks. It works nice and simple. That is
> really self sufficient. Org definitely not.
>
> HNB - Hierarchical Notebook
> http://hnb.sourceforge.net/
>
> In the mean time I have created database where I can store TODO,
> Notes, Calls, SMS, People, Invoices, Groups, Mailing Lists and so on,
> and made my own shell and Perl interfaces to it. And I used to manage
> it through: GeDaFe - PostgreSQL Generic Database Interface
> http://gedafe.github.io/doc/gedafe-sql.en.html and this was and is
> hierarchical or better graph knowledge management and relational
> database.
>
> Creating simple table in the database automatically helped me to
> manage that table. It is trivial to create NOTES table with schedule
> date, clock in, TODO or other conditions and tags. The interface is
> just there and automatic to whatever table I create the interface is
> there to add/modify/delete/search/refine records. That is what I would
> say "simple" and keeping things simple and indefinitely extensible
> without modification of software. The fundamental GeDaFe principle I
> would like to try to implement in Emacs. And same database I use for
> web interface I am using also within Emacs.
>
> GeDaFe principle is following:
>
> - define your data (like handling notes, TODO, or executing scripts within notes)
>
> - work and forget about any underlying functions,
>   add/create/delete/modify/index or search, make reports with simple exports
>
> - expand with more definitions of data when necessary (like add
>   various properties, or other data tables, like contacts, invoices,
>   etc.) and repeat the process.
>
> Org also shows that it does not hold "Notes" only, it holds more than
> that, I have written average book size technical documents with
> it. Only just one part of the document is printed from one single node
> that belongs to single project among many. People use such documents
> on the ground. My use case is not for simple notes.
>
> A node in a tree can be anything, and Org enhancements develop by that
> principle. For example there is org-contacts where nodes are meant to
> be "People". Such development is so much hard coded.
>
> Simpler would be to define the type of nodes and work by their
> types. One could need just one type table and one node table and that
> is about all.
>
> The type table could say:
>
> - this node is heading
> - or, this node is text under heading
> - or, this node is paragraph among many others
> - or, this node is hyperlink to WWW URI
> - or, this node is hyperlink to file
> - this node is movie to play
> - this type is PDF to be opened on page 3
> - this one is really note
> - this one is note but with action assigned like TODO, etc.
>
> Nodes could have its properties defined like for anything. Nodes can
> reference its parent nodes. Node can be parent to any heading.
>
> Once such 2 tables are defined magic starts to take place, it becomes
> its subtree with all kinds of node types including Babel execution and
> similar. Meta data is meta, it can be updated but need not be
> visible. Computer is handling meta data.
>
> Node can be a page in a subtree that becomes a website.
>
> Node can be object for person or company, or just anything.
>
> I am currently using my nodes to quickly research various subjects by
> using that type of dynamic knowledge repository.
>
> Org file is dynamic knowledge repository.
>
> About Dynamic Knowledge Repositories (DKR)
> https://www.dougengelbart.org/content/view/190/163/
>
> Then around 2015 I have discovered Cherrytree and have made bunch of
> notes with it, and that is self-sufficient one program that keeps
> simple things simple as it is much more simpler than Org. It offers a
> visual interface to all features related to the knowledge management
>
> Cherrytree - hierarchical note taking application with rich text and syntax highlighting
> https://www.giuspen.com/cherrytree/
>
> TAGS in Cherrytree are automatic if I remember well, TAG is simply
> name of the node. If I call node TODO, then nodes under are simply
> TODO items. 
>
> Later I discovered there is something similar in Emacs so I started
> with Org and use it mostly for instruction writing and project
> management. And I find many options over kill for me. On the other
> hand my usage of Org would be overkill for somebody, so it depends of
> viewpoints.
>
> In general I was always using headings and subheadings, trees, lists, notes by using text.
>
> Somewhere 2004 I started using Markdown one among first as I found it
> simpler than ASCIIDOC and M4, but not as perfect.
>
> 2020 and 2021 I like to raise the level of dynamic knowledge
> journaling to the meta level where I think less about underlying
> functions in software.
>
> That experience also tells that Org is definitely not simple.
>
> Among hnb, GeDaFe database approach, Cherrytree and Org mode, for
> "keeping things simple" like note taking and TODO items, project
> management, Cherrytree was the best for me.
>
> Among all those for keeping complex things simple the database
> approach is the best. Of course that I use database within Emacs and
> it is not visible to user what it is. Database allows simultaneous
> operation by several people on distance and that is the groupware
> feature as envisioned by Doug Engelbart.
>
> For document writing and nice formatting with LaTeX export, Org mode
> is the best personally.
>
> For notes, database based notes are best as only so I have relations
> between the note and other objects. With 200,000 contacts in the
> database which I can quickly access and assign notes to them, how
> would that work with Org? Hardly. Notes are related to people, to
> projects, plans, opportunities, research subjects, companies and so
> on. There is no simple way in Org mode to relate one note to multiple
> other related subjects.
>
> And people try to do exactly that, people are developing Org in the
> manner to relate objects in Org file to anything else. And they do
> hard work as they do it manually. Relational database speeds up and
> does not tell user to manually hyperlink various relations.
>
> I have sent thousands of tasks to people by using this function
> below. And I had to define for each Org file who are "assignees" for
> that Org file. And I have like 50 assignees, so I need to copy and
> paste their nick names or identifiers into the Org file. There it
> comes the attribute of being "self-sufficient", it becomes
> self-sufficient because I had to define all assignees into that
> specific file, but I find it tedious and not useful.
>
> (defun rcd/org-send-assigned-task ()
>   "Sends assigned task to designated individual as Org"
>   (interactive)
>   (let* ((member-data (rcd-org-extract-assigned-member-email-data))
> 	 (id (if member-data (first member-data) nil))
> 	 (signature (if (equal (type-of (symbol-value (fifth member-data))) 'cons)
> 	 		(third (symbol-value (fifth member-data))) ""))
> 	 (file (rcd-org-subtree-to-file signature))
> 	 (subject (rcd/org-find-headline))
> 	 (esubject (escape-% subject))
> 	 (ask (unless id (y-or-n-p "No assignment found. Do you want to send it by email?")))
> 	 (name (if id (third member-data)))
> 	 ;; (name (if ask (read-from-minibuffer "Name:") name))
> 	 (voice (format "The task '%s' is being sent to '%s'" subject name))
> 	 (email (if id (if (equal (type-of (fourth member-data)) 'cons)
> 			   (car (fourth member-data))
> 			 (fourth member-data))))
> 	 (email (if ask (cf-search-email (read-from-minibuffer "Search for email: ")) email))
> 	 (really (y-or-n-p (format "Do you really want to send it to: %s?" (if ask email name)))))
>     (if (and really (or id ask))
>       (if (string-match "@" email)
> 	(progn
> 	  ;; (message (escape-% subject))
> 	  (speak voice)
> 	  (mutt-send-file name email esubject file))
> 	(message "No email specified"))
>       (message "Aborted sending."))))
>
>> Moreover, some devs are even reluctant to hide metadata (like unique
>> ID implemented in org-id module) from users (which is possible and
>> also partially implemented).
>
> I hope that I have demonstrated that one who develops could review
> several various paradigms and learn more. I am fine with any decision
> of design for Org mode as I use it as it is and I have for me other
> ways of managing data. I am not sure how much those features have been
> discussed to say that hiding meta data is good or not good. It is
> better to discuss and find what is more useful.
>
> What I can see is that people complain for speed and they build
> extensions that help with it. Extensions are external while built-in
> Org does not keep up with the dynamic how people expect it to be.
>
> For example I would expect Org to be very simple, very very simple, I
> would rewind it back to its roots. Somebody else would like
> complexities. Currently I can see that Org is not made for
> complexities. It is better to unwind the development and make Org in
> core very basic and speedy and let people enhance it with external
> packages. In general it is better to remain simple. Even that may not
> be possible any more.
>
> I see hard work by many people who try to enhance Org as hierarchical
> knowledge of data because people want to have feature X, but group of
> those enhancements in reality belong to relational databases and not
> to text files. Developers wish to accommodate various people and yet
> by doing so they develop it into complex software. (129 .el packages
> for one Emacs mode!?)
>
> Among those one shall read the Doug Engelbart's paradigms, then
> especially if one is developer of system of data management that many
> people use, one shall explore other paradigms, including various
> approaches to data handling. And overall one shall keep it simple as
> it is main fundament of Org to be simple, while practical fact remains
> that it is not anymore simple, not at all. 
>
> I remember back in time with BASIC programming language that it had
> also something like a built-in database that one could put on the
> bottom of the file. Then there is also Perl's __DATA__ that is placed
> straight into the file and one could also place images and other
> stuff. Maybe the meta data could be kept this way in just one heading
> named Meta data, and this heading would not be exported, it could be
> hidden from the user and it could contain the database necessary for
> single Org file. By pressing key to show properties one could see
> properties or simply make them hidden. By making a copy of subtree the
> metadata would also copy as usual. By exporting, one could make Org
> without meta data, and I like using this information as I like sending
> Org headings without personal meta data to third parties.

Re: Is Org really so simple?

[Jean Louis]

* Ihor Radchenko <yantar92@gmail.com> [2020-11-26 06:09]:
>  Texas Cyberthal <texas.cyberthal@gmail.com>:
> > By philosophy, I mean the dev consensus on the correct way to do
> > things, and coded configuration and usability biases.
> 
> According to my experience with org-mode development (I am not talking
> about third-party packages here), it is discouraged to change org-mode
> towards hiding metadata in org files or store *unique* data (that cannot
> be derived from the contents of the org files) related to org-mode
> externally (not in org files). It is not official statement, but rather
> my impression so far.

Personally, and from the idea of having structured referencable
information I have derived my choice to move small pieces of
information to database management as that way I can keep unique IDs
without thinking and without having things sparse on the computer. I
am working on a meta level, editing pieces of information that are yet
to become Org file.

We use Org files for indexing and great chunk of code has been written
to help users index meta data, agenda does that for example. It tries
to make a main index. But it does not cope well with complex user
cases.

In my opinion Org could be more useful would there be more
improvements to collection of meta data from all of the system and
making it more available or exposed and accessible to
users. Multi-occur option in Agenda is nice and serves the same
purpose more or less and other options.

To explain "more exposed", that would mean that there could be easier
to access listing options that user can access the index in more
liberated way. The dialogue for agenda is somehow rigid, it gives
options to do this or that, one has to search or enter queries. But
that is where it stops.

When I say more liberated way, that is analogous to SQL query
language and I do not say users should learn something complicated.

Rather the query that is made by the user should be remembered as such
and be made available as a command, macro or similar that can be saved
for later.

Example is when I search for TODO entries with special TODO kwd, then
I may search for 3-4 keywords or 7-10 keywords during one
day. Then there would be repetition to search again for those
keywords. There is history value access in the minibuffer but that
does not remember searches for the next Emacs session. Having
rememberable customized searches for users would be useful.

Other issue with the Agenda window is that I cannot move from the
minibuffer to other window, as if I wish to enter TODO I may wish in
the same time to open some fiels and look for references while in the
same time I am working with the Agenda options. It is hard for me to
understand why it was made that way, could it be because of read-char
approach? 

> > And people try to do exactly that, people are developing Org in the
> > manner to relate objects in Org file to anything else. And they do
> > hard work as they do it manually. Relational database speeds up and
> > does not tell user to manually hyperlink various relations.
> 
> Could you provide some more examples? I do not see how relational
> database is different from creating hyperlinks in org. Either way, the
> user needs to file an object/headline somewhere into org file, which is
> inherently manual.

That is exactly how I am working personally and the software will be
available when polished. It is based on same principles of having
everything hyperlinked, stored, accessible, indexable, it is dynamic
knowledge repository which I use as for meta Org-like editing of
items. It can transform everything or just one subtree to Org files on
the fly which can be saved or delegated to others.

The explanation itself is in relational databases concept. We are
using Org mode and we already try doing, we are trying hard to make
relations stick together. Compared to SQL databases I see that as hard
work, effort to patch the unpatchable.

Maybe this URL could serve as good example to explain concept of
relational databases in simple manner:
https://www.dummies.com/programming/sql/knowing-just-enough-about-relational-databases/

Or more serious one from IBM (which probably does not explain concepts
well):
https://www.ibm.com/cloud/learn/relational-databases

If table in the database is related (defined or internally connected
by its meaning and values) to other tables, then user is spared of
writing repetitive tasks.

Practical example would be:

- if text file is internally related to Joe Doe, then by clicking on
  the text file such as Org file, I could automatically get various
  hyperlinks to anything related to Joe Doe: Joe's emails list, Joe's
  SMS list, Joe's contact information, I could click and initiate a
  call with my mobile phone and just write notes without thinking on a
  phone number, I could click in Org file and send SMS to Joe that
  will be saved in computer without thinking on Joe's phone number, I
  could see relatives of Joe and find his sister and again have all
  the hyperlinks and relations to various other pieces of information
  related to Joe.

- and if I am in an Org file that has relation to other objects I
  would not need to construct hyperlinks by hand, they would be or
  could be constructed semi-automatically because the relation is
  already known.

Quoting me:

> > And people try to do exactly that, people are developing Org in the
> > manner to relate objects in Org file to anything else.

One example of way too much work is making unique IDs. Either user
decides to make it by invoking M-x, or saves the file and uses hook,
or does not invoke it. So it is in the Org but subset of users does
need Org ID numbers.

When I work with the database and wish to edit something equivalent to
a headline than the unique ID is generated automatically and I do not
need to think of it. It is made by database design.

Excerpt from hlinks table on my side:

CREATE TABLE hlinks (
hlinks_id SERIAL NOT NULL PRIMARY KEY,
....

Instead of customizing Emacs to insert IDs, customizing Org,
customizing specific files, going over specific files to find which
has unique ID which one does not have, having duplicates all over the
place, having pretty ugly and not meaningfull unique ID always shown
on screen and more or less to a degree disturbing, instead of having
them easily editable and changeable and error prone, instead of many
actions related to that unique ID, I have the above one line that does
it for me as database does the work. I need not think any more and
since I created that table I did not think until today about the
unique IDs because they are very reliable, stable, forever, database
is thinking for me, computer is taking job from me, and I am not
working for my computer.

Now if unique ID is related to one of following other tables of
subjects, objects:

- last USER who ever modified the heading, note, task, URL, anything
  related to that ID, I can find that last user. Hyperlink could be
  automatically generated button that points to on the fly generated
  Org buffer showing me such modification.

- I can find ANY USER who ever modified anything related to that
  unique ID and I can find specific time of modification in past, with
  hyperlink that I do not make by hand, that would show me all users
  and from which list I could hyperlink to the first example above.

- There is DATE in the hlinks table, so it is possible to have date as
  hyperlink button from where I can click to find other same date
  related entries, or I could invoke list to find nearby dates.

I do not create hyperlinks more than once in a function:

  (defun hyperscope-insert-buttons (tags)
  (let ((tags (split-string tags)))
    (insert "   Tags: ")
    (dolist (tag tags)
      (hyperscope-insert-button tag)
      (insert " "))))

  (defun hyperscope-insert-button (tag)
  (insert-text-button tag
		      'action
		      `(lambda (b) (hyperscope nil nil nil ,tag))
		      'mouse-action
		      `(lambda (b) (hyperscope nil nil nil ,tag))))

  That is my button. I never create those hyperlinks again. This
  relates only to the tags, but hyperlinks can be for any other type
  of relation.

  If the tag is "Dewi" the hyperlink is automatically shown in the
  header of the entry. It allows me to quickly see all entries for
  "Dewi" tag and come back to it.

- the heading of meta Org can have a status defined by user anyhow and
  I do not mean TODO/DONE but could be anything that changes later
  actions of the user, or automatically generates hyperlinks.

- possible EXPIRATION entry is there, it could be automatically
  hyperlinked to other expired items

- CURATOR entry is there, which is multi user based. User writing in
  the database is not necessarily curator of the entry. All items to
  curator's lists can be shown by hyperlinking automatically.

- HYPERLINK TYPE could be Message-ID to open up specific email
  message, then I could get a hyperlink to open all Message-IDs by
  that attribute

- MIME TYPE is there as if it is text/org then database knows it will
  invoke for me the Org mode. There will be no need for local
  variables, although it could be implemented. If there are various
  mime types the hyperlink can be there for user to list them all.

- HYPERLINK NAME is hyperlinked itself and can be hyperlinked and
  shown from one section to other just as Org mode. Org mode in fact
  can be used to display those pieces of information.

  Discussion here is only there to give ideas to me and others to
  lessen repetitive actions and help us in creating more useful
  features.

- HYPERLINK itself is where user goes. On my side Hyperlinks are
  centralized and this is very handy as I do not repeat myself to edit
  and change numerous files when creating hyperlinks in such
  files. Person does not need a PostgreSQL database to create these
  kind of universal hyperlinks, it just needs to know one centralized
  file. It can be defcustom option in Emacs. But it could be editable
  list. Or it could be simple text file with unique IDs at front and
  description.

  Example of such file would be:

  1 GNU Website, free software for you @@@ https://www.gnu.org
  2 A new skimmer uses WebSockets and a fake credit card @@@ https://blogs.akamai.com/2020/11/a-new-skimmer-uses-websockets-and-a-fake-credit-card-form-to-steal-sensitive-data.html

It is trivial to have a function that is called my-link which would
then parse the file, find the ID number at front and have the user
jump to the link. User could be offered file with links to "store"
link or transfer straight to Org file.

Org file would have the link something like:

[[(my-link 1)][GNU Website, free software for you]]

If the underlying hyperlink changes for some reason such hyperlink
need not be changed in Org files.

What if hyperlink points to specific PDF file on the system? Why
should I hard code in the sparse disconnected Org file a hyperlink to
specific file on the file system? Maybe this file changes its
position, then my hyperlink in Org file will not work, what if same
hyperlink has been used 10, 20 or 50 times in other Org files?

When using centralized approach for hyperlinks you forget about
editing the Org files, you just edit the centralized list of such
hyperlinks. That is what I am doing now.

Yesterday I have implemented new feature based on what we were
speaking about filing files and I have now made redundant long time
filing system. Database has sets, set is like a parent of a subtree or
subtree name. I have connected sets by their unique IDs to the
specific directory on the file system. Directory is based on the ID
number. Maybe you noticed that I file files related to specific person
to their directory on the hard disk, but I never think of the
directory position. In Dired I am invoking s-f and I am asked for
person's name, I choose the person and files are filed there by
year/month/date or otherwise or I can choose the date.

Then we discussed how some people access files by using searches or
indexed databases, searching by terms, queries, tags, and there is
also way to access files by navigating file system what most people
do. And then I mentioned conceptual access. This last one accommodates
better my mind and I was already used to file files by people,
organizations and subject.

From there now I just think of a file and file it under specific
subject.

If I think of specific subject, I can access the file. If I think of
related subjects I can still find somehow the file.

For example I think of LISP something, nothing specific, press key,
type LISP maybe there are 63 entries, but one only is upper case I am
on the LISP entry displayed in Emacs within a second. This entry is a
set or parent of subtree of other sets. I could now quickly jump into
the LISP directory:

The entry (ID numbers need not be shown):

    35147 LISP                                                                   Set →

Now I could jump from that entry into Dired:

  /home/data1/protected/hyperscope/3/5/1/4/7:
  
And the directory ID remains always the same /3/5/1/4/7 regardless if
I modify somewhat the entry. And if I enter the Set or subtree related
to LISP then I could get other entries such as CLISP, Emacs Lisp,
Newlisp, Picolisp and so on.

If I file something in those directories, I could invoke indexing
option to curate some of those sub-directories within
/home/data1/protected/hyperscope/3/5/1/4/7

While this looks complicate when reading, practically it is easier to
file files under specific subjects or categories this way, as this is
the workflow:

- open Dired
- think where you wish to file
- press key to choose what you thought like "LISP"
- press enter

No need to think of the directory as directory has been choosen
automatically.

Next time I need to jump to directory:

- think where to jump
- press key to find the meaning, and ENTER

Back to hyperlinks in Org file. If I am writing an entry in meta
Org-like manner then hyperlinks are generated automatically that go to
related files, that goes to related other subjects.

> Could you provide some more examples? I do not see how relational
> database is different from creating hyperlinks in org.

I am already using this system but is not easy to explain. Meta data
on my side is not hidden but is also not shown. I can edit it as easy
as Org probably easier.

c a - I am editing author's name in free text. If author is in the
      database I would choose author. Author's name can be hyperlinked to
      find other entries under same author. No need to type it by hand.

T - I am editing tags freely, I can enter: emacs lisp todo review; and
    those words would become tags. After ENTER I can immediately see
    hyperlinks to those tags.

Then if I am to edit some special status or type of the node I get
also automatic hyperlinks.

>  Either way, the
> user needs to file an object/headline somewhere into org file, which is
> inherently manual.

Yes, and no. Yes because it is true, and no because Org has been
already developed to help the user and useful enhancements or not
limited.

Concepts I have explained above need not be used with a database.
Concept is of relations and that user would not need to repeat many
actions if concepts are pulled together and integrated. 

Concept of using centralized list would alone spare so much typing for
the user.

How it would be if some headlines are repetitive? Maybe centralized
list of headlines to be choosen from and quickly inserted would help
speed up the process. Deadline would maybe remain the same or other
properties. No need for tedious searching for headling, duplicate, or
copy and yank the headline again. Choose from the central list and
build on that.

Headlines are related to things like people, company, organizations,
and those descriptions of companies, people, etc. majority of Org
users probably keep in the Org file itself. I just think it could be
so. But it is bunch of related information, including files.

If headline is related to central list of IDs that relate to something
then user could automatically invoke or get simply visualized
hyperlinks, or by click decide to insert such.

Central list would have entries like these:

1 Joe Doe @@ ~/files/JoeDoe @@ +1 234 567-8900 @@ joedoe@example.com @@ etc.

Then simple function can construct various hyperlinks for the headline
that has specific property.

Let us say I wish to insert new headline related to Joe Doe in such
imaginary enhancement:

- press key
- choose Joe Doe by maybe completion functions (this prepares property)
- headline is prepared and I just write it while property remains there

Prepared headling could look like this with X being my cursor:

* X                                 :CID-1:

Now I write the headline:

* Joe to purchase ticket in Kahama and travel on Saturday to Dar es Salaam  :CID-1:

If the task is assigned to Communication Officer or senior who has to
advise Joe how to do it, then on the next key press I could get
various hyperlinks because there is CID-1 that relates to Joe Doe:

* Joe to purchase ticket in Kahama and travel on Saturday to Dar es Salaam  :CID-1:

  [Joe's files] [Initiate call] [SMS to Joe] [Send this task to Joe] [Plain email to Joe]

Or the heading can automatically become following

* Joe to purchase ticket in Kahama and travel on Saturday to Dar es Salaam  :CID-1:

-  Joe Doe phone number: +1 234 567-890
-       Joe Doe's files: ~/files/JoeDoe
-       Joe Doe's email: joedoe@example.com
-        Family members: [Sister] [Brother] [Father]
- Other Joe Doe's tasks: [All tasks] [TODO Tasks]
-     People introduced: [173 people introduced]
-              Location: [Show on the map]
-          Transactions: [Show table of transactions]

Or I could just enter some of those hyperlinks specifically. But maybe
when I am choosing heading with the relation CID-1 property I do wish
to have some of those links, or I could prepare the template or
customize how those hyperlinks would look like.

> > I see hard work by many people who try to enhance Org as hierarchical
> > knowledge of data because people want to have feature X, but group of
> > those enhancements in reality belong to relational databases and not
> > to text files.
> 
> It is an interesting point. I would be happy if some existing tools
> could be reused instead of re-inventing the wheel for org. Do you have
> concrete examples where it can be useful? If you have, I encourage you
> to bring up a feature request to discuss this possibility with org-mode
> devs.

By principle concepts and relational automation does belong into core
of the Org mode as Org as of today does follow principles of Doug
Engelbart.

Doug Engelbart Institute - Boosting mankind's capability for coping with complex, urgent problems
https://www.dougengelbart.org/

Draft OHS-Project Plan
https://web.archive.org/web/20070704185333/http://www.bootstrap.org/augdocs/bi-2120.html

TECHNOLOGY TEMPLATE PROJECT OHS Framework 
https://www.dougengelbart.org/content/view/110/460/

About Dynamic Knowledge Repositories (DKR)
https://www.dougengelbart.org/content/view/190/163/

Or maybe you wish to see the Mother of all Demos.

Highlights of the 1968 "Mother of All Demos"
https://www.dougengelbart.org/content/view/276/000/

As there you will see the outline mode hyperlinked to everything being
hyperlinked from anything else, which principles are partially
implemented on the same website, you can see how finely grained one
can reference a paragraph on that website.

Maybe in beginning the Org mode was just enhancement of the Outline
mode which I find great by the way for its simplicity. But today Org
mode has evolved into sparse or disconnected way of doing
things. Developers try their best. In my opinion from somebody who
uses relational tables I do not find that approach nice, I find it
very tedious and time spending activity.

I would like to see Org file being finely grained so that each item of
the list can be referenced, each paragraph to be uniquely referenced
or sets of paragraphs. There is a package that I guess, converts Org
file to OPML which is an acronym for Outline Processor Markup
Language, as Org is outline processor that tries doing it in plain
text. It misses all the meta data so tendencies by people go back to
more structured format.

Because specific paragraph referencing is now not easily possible I am
using database approach of editing chunks of the Org file where then
any piece of Org can become referential.

Maybe the fundamental feature in its basics already exists if the
whole Org file could be exported as LISP data structure. Then user
could say "chunk paragraphs" to get references to single
paragraphs. Or feature could run over the Org file and create unique
IDs or anchors for each paragraph which could then be referenced in
their exports. If Org is exported to HTML then referencing from other
documents to specific paragraph becomes possible. While this is my
wish, it is not necessarily wish for Org.

GNU Hyperbole
https://www.gnu.org/s/hyperbole

The GNU Hyperbole Koutline mode predates Org mode and allows finely
grained referencing. But it is not really plain text. For example
there are no headings or paragraphs as everything is a structured
node. This may not be so handy. Mixture of Koutliner and Org mode is
what I like to build myself. If I work on meta Org-kind-of level then
I am editing nodes and for those nodes I am deciding what they are,
maybe node is a Set or Heading, maybe it is WWW hyperlink, maybe it is
note, maybe single paragraph to be processed as such and formatted as
such in various outputs, maybe it is set of paragraphs, and maybe it
is one chunk of Org file or any other type of file, could be even
image, video, anything. Then when there are chunks defined I can
structure the output or move nodes up and down with easy.

To create Org file from meta Org-like editing is speedy and not
noticable. If one set of chunks of such Org file have been used in
other set or vice versa, I can use reuse those nodes without
duplicating the node. If specific table of transaction is referenced
from multiple files users of Org will want to include it in those
files. That is also what minimizes replication, I do not need to
duplicate the body or meta data or contents of elements but I just
need to say what is the series by semi-visual ordering of those
elements by their priority.

Heading A can be one node, edited by user one time. It need not be
duplicated for other file, rather only a pointer to Heading A can be
included and the Heading then appears in a new Org file.

This type of link here is fine for plain text, really makes sense:

** TODO Some heading here

It just does not scale and does not have relations. As there are no
automatically designated relations then I have to assign it myself:

- to which person this task is assigned? Normally I send tasks to
  person assigned

- which person is related to this task? Do I really need to write the
  name? Then I have no reference or relation back to the person. I
  would like to have but do not. My tasks are "For staff Joe to go to
  location ABC and negotiate with company XYZ".

  If task is related to company which can be just on second to relate
  it, and assigned to staff Joe, which is another second or few to
  find Jow, and location ABC which is another few seconds and related
  to company XYZ which is another few seconds then all what I do are
  selections. No hyperlink writing on the low level.

  The task that is assigned to Joe, then can be sent to Joe with all
  the relevant and related information such as geographic location of
  the location ABC, license number related, region or district where
  it is located, then all contact information related to Joe and
  contact information related to company XYZ, including people names
  related to company XYZ and their contacts. My time is saved. Joe has
  got full information and can fullfil the task. This may include the
  scheduled date and deadlines as well and it could also include list
  of related tasks, without me thinking. 

To say TODO, PENDING, add property this or that by hand, is repetition
that I am not inclined any more to perform, so I am transitioning
everything into meta level. Using few keys I can export it to Org and
use further Org facilities to export into PDF or ASCII, Unicode or
similar.

Org started as enhancement of the outline-mode with focus on plain
text. But it is not any more plain text, it is full of properties that
may not be readable to people who did not write them or who are not
used to it. Properties can also confuse the person and performance of
task can be ruined. So it develops more and more into activity that
does need relational database. If such relational database is
implemented in the Emacs EIEIO or maybe as more simpler LISP data or
various lists of hashes saved in files, or is maybe PostgreSQL or
maybe GDBM or MySQL database, NoSQL, that really does not matter.

What matters is that headings are related to various lists of other
things and users do not have relational facilities. They write them
tedious by hand and developers try to help by hacking something that
has been already invented. You said let us not reinvent the wheel but
Org mode is wheel reinvented regardless if it did not use Engelbart's
principles it became quite similar to Augment as outliner software.

Doug Engelbart Institute - About NLS / Augment - 1968
https://www.dougengelbart.org/content/view/155/87/

Re: Is Org really so simple?

[Ihor Radchenko]

Jean Louis <bugs@gnu.support> writes:
> Rather the query that is made by the user should be remembered as such
> and be made available as a command, macro or similar that can be saved
> for later.

Users can explicitly define custom searches using
org-agenda-custom-commands. Though format of that variable is indeed
quite complex (similar to org-capture-templates).

> Example is when I search for TODO entries with special TODO kwd, then
> I may search for 3-4 keywords or 7-10 keywords during one
> day. Then there would be repetition to search again for those
> keywords. There is history value access in the minibuffer but that
> does not remember searches for the next Emacs session. Having
> rememberable customized searches for users would be useful.

Take a look at savehist-mode. Enabling it should save all the minibuffer
history by default. However, the history of your agenda searches will be
mixed with global minibuffer history. If you do not like this behaviour,
feel free to open feature request to make org-agenda keep history
separately. Then, savehist can be configured to save that across Emacs
sessions.

> Other issue with the Agenda window is that I cannot move from the
> minibuffer to other window, as if I wish to enter TODO I may wish in
> the same time to open some fiels and look for references while in the
> same time I am working with the Agenda options. It is hard for me to
> understand why it was made that way, could it be because of read-char
> approach? 

See my other reply. When you are actually entering TODO search, you can
switch from minibuffer to other windows as usual.

> - if text file is internally related to Joe Doe, then by clicking on
>   the text file such as Org file, I could automatically get various
>   hyperlinks to anything related to Joe Doe: Joe's emails list, Joe's
>   SMS list, Joe's contact information, I could click and initiate a
>   call with my mobile phone and just write notes without thinking on a
>   phone number, I could click in Org file and send SMS to Joe that
>   will be saved in computer without thinking on Joe's phone number, I
>   could see relatives of Joe and find his sister and again have all
>   the hyperlinks and relations to various other pieces of information
>   related to Joe.
>
> - and if I am in an Org file that has relation to other objects I
>   would not need to construct hyperlinks by hand, they would be or
>   could be constructed semi-automatically because the relation is
>   already known.

What you are describing appears to be easy to achieve in org files if
one wishes to:

some_file.org:

#+begin_src org
Lorem ipsum dolor sit amet, consectetuer adipiscing elit. Donec
hendrerit tempor tellus. Donec pretium posuere tellus. Proin quam nisl,
tincidunt et, mattis eget, convallis nec, purus. Cum sociis natoque
# clicking on the link with bring you to Joe Doe's heading in Joe Doe.org
[[id:joe-id][Joe Doe]]
penatibus et magnis dis parturient montes, nascetur ridiculus mus. Nulla
posuere. Donec vitae dolor. Nullam tristique diam non turpis. Cras
placerat accumsan nulla. Nullam rutrum. Nam vestibulum accumsan nisl.
#+end_src

Joe Doe.org:

#+begin_src org
# C-c C-o at the heading will show all the links and allow "clicking"
# one of them
# C-c C-j will allow free-hand selection of any sub-heading
# C-s will allow free-hand search across the file
,* Joe Doe :person:
:PROPERTIES:
:ID: joe-id
:END:

Contact information:
100 Street, City, Country
# Clicking the following link would start the phone/skype call
Tel: [[call:11111111111][call to 11111111111]]
[[sms:111111111][send sms to 111111111]]

,** SMS messages
,*** [[id:message1-id][Message 1]]
,*** [[id:message2-id][Message 2]]

,** Related
,*** [[id:antony-id][Antony Joe]]
,*** [[id:something][Something else]]
,*** [[id:yetmore][Yet another one]]
#+end_src

> When I work with the database and wish to edit something equivalent to
> a headline than the unique ID is generated automatically and I do not
> need to think of it. It is made by database design.

Note that not all users would want this by default. But you can simply
set org-id-link-to-org-use-id to 't and the unique ids will be created
every time you create a link to heading.

> Instead of customizing Emacs to insert IDs, customizing Org,
> customizing specific files,

It is an advantage for me. Not all the people need the specific workflow
you like, but it can thankfully be changed. Same as any other part of
Emacs.

> ... going over specific files to find which
> has unique ID which one does not have, having duplicates all over the
> place,

Why do you need to find it yourself? It is all handled by org.
Duplicates are rare and mostly happen in archives/trash according to my
experience.

> ... having pretty ugly and not meaningfull unique ID always shown
> on screen and more or less to a degree disturbing,

org-custom-properties ;) 

> ... instead of having
> them easily editable and changeable and error prone,

Agree. This is the price of keeping everything in plain text.

> ... instead of many
> actions related to that unique ID,

Sorry, I do not understand which actions you refer to.

> ... I have the above one line that does
> it for me as database does the work. I need not think any more and
> since I created that table I did not think until today about the
> unique IDs because they are very reliable, stable, forever, database
> is thinking for me, computer is taking job from me, and I am not
> working for my computer.

So, database clearly works better for your specific workflow. I
personally, do not mind text-based IDs all that much.

> - last USER who ever modified the heading, note, task, URL, anything
>   related to that ID, I can find that last user. Hyperlink could be
>   automatically generated button that points to on the fly generated
>   Org buffer showing me such modification.

I do not understand how you determine the user editing something
in the database. Or do I misunderstand?

> - There is DATE in the hlinks table, so it is possible to have date as
>   hyperlink button from where I can click to find other same date
>   related entries, or I could invoke list to find nearby dates.

FYI. Clicking on timestamps in org automatically invokes agenda view
listing all the headings containing timestamps from the same day.

>  That is my button. I never create those hyperlinks again. This
>  relates only to the tags, but hyperlinks can be for any other type
>  of relation.

FYI. When you have multiple tags in your heading, clicking on the tags
will automatically show agenda view listing all the headings with
matching tags.

> - the heading of meta Org can have a status defined by user anyhow and
>   I do not mean TODO/DONE but could be anything that changes later
>   actions of the user, or automatically generates hyperlinks.

The keywords in org are not limited to TODO/DONE. One can create
arbitrary set of keywords (representing the status) and even define them
locally in buffer (see "5.2.5 Setting up keywords for individual files"
in the manual).

> - possible EXPIRATION entry is there, it could be automatically
>   hyperlinked to other expired items

Generally, any database field can be represented in org using property
drawers (note that you can even set global property drawer at the top of
the org file before the first heading). For the expiration specifically,
there is even a package in org - org-expity ;)

> - HYPERLINK TYPE could be Message-ID to open up specific email
>   message, then I could get a hyperlink to open all Message-IDs by
>   that attribute

Similarly, you can hyperlink anything from org-mode. Emails are no
exception.

>  Discussion here is only there to give ideas to me and others to
>  lessen repetitive actions and help us in creating more useful
>  features.

I am not sure if it is going to be useful for creating features.
However, it indeed generated many insights how one can deal with
personal information management using org-mode.

Some things may not be very clear for an arbitrary reader though. For
example, it took me quite a lot of time to wrap my head around the idea
of using databases - I was thinking too much in terms of org-mode files.
Would you consider making a blog post showing your workflows in more
consistent way with examples and possibly gif demonstrations?

> What if hyperlink points to specific PDF file on the system? Why
> should I hard code in the sparse disconnected Org file a hyperlink to
> specific file on the file system? Maybe this file changes its
> position, then my hyperlink in Org file will not work, what if same
> hyperlink has been used 10, 20 or 50 times in other Org files?

I personally approach this by keeping my files as attachments to org
headings. Then, the files can be references simply by referencing the
relevant heading (equivalent of your record in the database).

> - open Dired
> - think where you wish to file
> - press key to choose what you thought like "LISP"
> - press enter

Have you ever been in scenario when you want to file something to
multiple locations?

> c a - I am editing author's name in free text. If author is in the
>       database I would choose author. Author's name can be hyperlinked to
>       find other entries under same author. No need to type it by hand.

Do you need to set the link to author every single time you create a new
entry authored/related by/to that author? Or do you somehow create the
link to the author automatically?

> T - I am editing tags freely, I can enter: emacs lisp todo review; and
>     those words would become tags. After ENTER I can immediately see
>     hyperlinks to those tags.

Are your tags also uniquely defined using relational database? Say, you
want to change some tag name for some reason. Will changing the name
automatically update all the tags in the entries marked with that tag?

> If the task is assigned to Communication Officer or senior who has to
> advise Joe how to do it, then on the next key press I could get
> various hyperlinks because there is CID-1 that relates to Joe Doe:
> 
> * Joe to purchase ticket in Kahama and travel on Saturday to Dar es Salaam  :CID-1:
> 
>   [Joe's files] [Initiate call] [SMS to Joe] [Send this task to Joe] [Plain email to Joe]

This is an interesting idea. You mentioned it in the past, but this
example made it much more clear.

Something similar is implemented in org-ref
(https://github.com/jkitchin/org-ref). Instead of adding links like
[Joe's files] directly into the text, org-ref shows various
possibilities when one clicks on the link itself - there is a completion
dialogue suggesting actions related to the link. Since org-ref is
specifically designed for referencing scientific articles, it implements
only a single set of suggestions like: (1) open pdf of the article; (2)
show notes; (3) email article to someone; (4) open url page of the
article, ...

> Because specific paragraph referencing is now not easily possible I am
> using database approach of editing chunks of the Org file where then
> any piece of Org can become referential.

It is actually possible to reference arbitrary chunks of text in org:

"4.2 Internal Links" section of the manual:

     1. one item
     2. <<target>>another item
     Here we refer to item [[target]].

Or using #+NAME: keyword

     #+NAME: My Target
     | a  | table      |
     |----+------------|
     | of | four cells |

Unfortunately, it only works inside a single file. Though I imagine that
org-id can be modified to support global referencing.

You can send a feature request about this if you are interested.

> Heading A can be one node, edited by user one time. It need not be
> duplicated for other file, rather only a pointer to Heading A can be
> included and the Heading then appears in a new Org file.

That would be a great feature indeed. This idea is actually getting a
lot of attention recently:
https://github.com/alphapapa/transclusion-in-emacs

Best,
Ihor

Re: Is Org really so simple?

[Jean Louis]

* Ihor Radchenko <yantar92@gmail.com> [2020-11-29 10:16]:
> 
> Jean Louis <bugs@gnu.support> writes:
> > Rather the query that is made by the user should be remembered as such
> > and be made available as a command, macro or similar that can be saved
> > for later.
> 
> Users can explicitly define custom searches using
> org-agenda-custom-commands.

That is definitely useful feature. Just not as accessible and
understandable. It needs chunk of effort to learn. I would rather
prefer something like:

"I wish to find [Headings] in [file] from [date] to [date] etc"
                [TODO Headings]
		[TAGS]
		[Headings with tags]
                etc.
		^^^^^^^^^^^^^
		drop down

Not for me, but for future of Org mode I would find dropdowns more
useful for users who would simply say what they want and choose it
from drop downs "I wish to get list of ....... with attributes
......."

And multiple such examples could be offered and user could customize
them. As C in (org-agenda) function shows this:

Hide Org Agenda Custom Commands:
INS DEL Choice: Value Menu Command series, all agenda files:
            Access Key(s): n
            Description  : Agenda and all TODOs
            Component:
            INS DEL Choice: Value Menu Agenda:
                        Local settings for this command.  Remember to quote values:
                        INS
            INS DEL Choice: Value Menu TODO list (all keywords):
                        Local settings for this command.  Remember to quote values:
                        INS
            INS
            Settings for entire command set:
            INS
            [ ] Export:
                INS

Which is simply not user friendly. I would prefer it as:

"I wish to find [Headings] in [file] from [date] to [date] etc"
                [TODO Headings]
		[TAGS]
		[Headings with tags]
                etc.
		^^^^^^^^^^^^^
		drop down

where [date] is not important but generation and construction of a
human readable expression that user may easier relate to it.

INS - what is it? Sure I know, but user only wants to see
agenda. Emacs is advanced editor but we invite non-advanced users to
become advanced. Those advanced are already using it.

This design being too hackish be major block for an organization to
adopt Emacs without previous knowledge about it.

            INS DEL Choice: Value Menu Agenda:
                        Local settings for this command.  Remember to quote values:
                        INS
            INS DEL Choice: Value Menu TODO list (all keywords):
                        Local settings for this command.  Remember to quote values:
                        INS

Agenda in the real world is never that complicated and never
complicated in same difficult manner.

Paper based organization or even hand writing is easier than using the
Org mode. It is for advanced users and it would be great to see
simplifications and integrations bringing it closer to beginners.

In comparison to Emacs based agenda views, I find it easier to use
SQL:

SELECT notes FROM table WHERE notes_tags LIKE 'TODO';

Something like that is more readable and human friendly.

I find it funny that in customizations somebody actually put effort to
tell users "Remember to quote values:".

> Though format of that variable is indeed quite complex (similar to
> org-capture-templates).

I think that wizard or code generation function is very easy to make
in Emacs, think like this:

- press C or other key to generate your own query
- User is asked "Press a key you wish to use for this query?"
- User press "n"
- Then user is offered values in a visible way to choose as Choice
   such as those in customization
- "Do you want to add more choices to search from? yes/no"
- "Do you want to export?"

etc.

When designing computer programs one should think to make things
faster, easier, more useful than what people would otherwise do let us
say on paper. If paper tasks have been spread in various groups, those
would be already containing headlines that may be used for agenda.

> > Example is when I search for TODO entries with special TODO kwd, then
> > I may search for 3-4 keywords or 7-10 keywords during one
> > day. Then there would be repetition to search again for those
> > keywords. There is history value access in the minibuffer but that
> > does not remember searches for the next Emacs session. Having
> > rememberable customized searches for users would be useful.
> 
> Take a look at savehist-mode. Enabling it should save all the minibuffer
> history by default. However, the history of your agenda searches will be
> mixed with global minibuffer history. If you do not like this behaviour,
> feel free to open feature request to make org-agenda keep history
> separately. Then, savehist can be configured to save that across Emacs
> sessions.

savehist-mode is all time enabled on my side. I am not sure if I can
see here in the function `completing-read-multiple' that it does uses
history.

        (setq org-select-this-todo-keyword
              (mapconcat #'identity
                         (let ((crm-separator "|"))
                           (completing-read-multiple
                            "Keyword (or KWD1|KWD2|...): "
                            (mapcar #'list kwds) nil nil))
                         "|")))

The short function's definition is here:

(completing-read-multiple PROMPT TABLE &optional PREDICATE
REQUIRE-MATCH INITIAL-INPUT HIST DEF INHERIT-INPUT-METHOD)

and 6th element should be history variable. I do not see the 6th
element there and that is why it is not remembered. It uses history
from anything, but not history specifically to Org files.

> > - if text file is internally related to Joe Doe, then by clicking on
> >   the text file such as Org file, I could automatically get various
> >   hyperlinks to anything related to Joe Doe: Joe's emails list, Joe's
> >   SMS list, Joe's contact information, I could click and initiate a
> >   call with my mobile phone and just write notes without thinking on a
> >   phone number, I could click in Org file and send SMS to Joe that
> >   will be saved in computer without thinking on Joe's phone number, I
> >   could see relatives of Joe and find his sister and again have all
> >   the hyperlinks and relations to various other pieces of information
> >   related to Joe.
> >
> > - and if I am in an Org file that has relation to other objects I
> >   would not need to construct hyperlinks by hand, they would be or
> >   could be constructed semi-automatically because the relation is
> >   already known.
> 
> What you are describing appears to be easy to achieve in org files if
> one wishes to:
> 
> some_file.org:
> 
> #+begin_src org
> Lorem ipsum dolor sit amet, consectetuer adipiscing elit. Donec
> hendrerit tempor tellus. Donec pretium posuere tellus. Proin quam nisl,
> tincidunt et, mattis eget, convallis nec, purus. Cum sociis natoque
> # clicking on the link with bring you to Joe Doe's heading in Joe Doe.org
> [[id:joe-id][Joe Doe]]
> penatibus et magnis dis parturient montes, nascetur ridiculus mus. Nulla
> posuere. Donec vitae dolor. Nullam tristique diam non turpis. Cras
> placerat accumsan nulla. Nullam rutrum. Nam vestibulum accumsan nisl.
> #+end_src
> Contact information:
> 100 Street, City, Country
> # Clicking the following link would start the phone/skype call
> Tel: [[call:11111111111][call to 11111111111]]
> [[sms:111111111][send sms to 111111111]]
> 
> ,** SMS messages
> ,*** [[id:message1-id][Message 1]]
> ,*** [[id:message2-id][Message 2]]
> 
> ,** Related
> ,*** [[id:antony-id][Antony Joe]]
> ,*** [[id:something][Something else]]
> ,*** [[id:yetmore][Yet another one]]
> #+end_src

Thank you for pointing that out. You are very right that it is
possible. Only it is tedious without users' functions to accomplish
the task easier. Phone numbers are typed, not that heading is related
to the phone number. Typing is error prone. Hyperlinks need so much
more developed for users to automatically create them. Something like
open menu, insert phone number as hyperlink. No thinking about which
phone number, which person. If person's ID is already there, computer
should find it.

I am aware that by hand work one can construct such relations as you
have demonstrated, just that it is hard work in the end.

> > When I work with the database and wish to edit something equivalent to
> > a headline than the unique ID is generated automatically and I do not
> > need to think of it. It is made by database design.
> 
> Note that not all users would want this by default. But you can simply
> set org-id-link-to-org-use-id to 't and the unique ids will be created
> every time you create a link to heading.

That is true and not all people use it. I was used to it from database
work. I will soon abolish them all and move tasks to meta level
including notes, then when I need Org to be distributed I can export
in a second to Org file.

> > Instead of customizing Emacs to insert IDs, customizing Org,
> > customizing specific files,
> 
> It is an advantage for me. Not all the people need the specific workflow
> you like, but it can thankfully be changed. Same as any other part of
> Emacs.

That is like bicycle and motorcycle, as bicycle is advantage for
walkers but motorcycle for bicycle drivers. We can observe people
having problems with execution of functions, speed, queries,
customizations, etc. The design of Org mode in its root as plain text
is very fine and good. But when it jumped into management of
structured data that is where Org mode became something different. Now
the strategy of Org mode is to hard code and to provide functions that
databases provided for decades.

When user edits its file (Org), after a while there is nice
enlightenment, satisfaction and pleasure. This is because it helps to
keep things organized. But try to share it with your fellows, friends,
family who do not know about Org mode. And then is good to draw
conclusions from difficulties they are faced to as those will be
speaking of usability.

> > ... going over specific files to find which
> > has unique ID which one does not have, having duplicates all over the
> > place,
> 
> Why do you need to find it yourself? It is all handled by org.
> Duplicates are rare and mostly happen in archives/trash according to my
> experience.

I have 50 duplicates now. And it comes from copy and paste of
subtrees. As user I am warned to look into *Messages* buffer to help
me. Then I find there:

Duplicate ID "2759bfcf-73fb-4f8a-9a44-04553797185e"
Duplicate ID "b981c204-ecf7-41a3-a6f4-f40c22f6823f"
Duplicate ID "8e507f2b-8c43-46d0-903f-e667c914e358" [2 times]
Duplicate ID "c1565202-9f74-4c70-91b6-3e7bf55af802" [2 times]

which tells me nothing. Do not mind these thoughts for me personally,
I am fine. These thoughts describe where Org mode need enhancement.

When I was making my own functions it was trivial to make for example
separate buffer where messages from async-shell-command are displayed
so that I can easier access them. They are not scattered all over
not-related messages in the put-it-all buffer named *Messages*. Think
about usability for others, not for me. I am transitioning from Org
and doing it on meta level. So I am liberated from various designs
that do not help me.

That there is possibility to have duplicate ID is already enough for
me. I like CREATE TABLE id SERIAL NOT NULL PRIMARY KEY and finished. I
never even think of ID numbers, they are made for me, stable, and
always unique.

Back to {M-x org-id-update-id-locations} as that function should or
could open up the separate buffer and use the Org mode in that file to
construct hyperlinks to those duplicate IDs. Why is Org mode function
like {M-x org-id-update-id-locations} using *Messages* buffer to
display errors is very hard for me to understand. It breaks all my
logic down. Why I do not get any relation from the duplicate ID to the
file where it should be found?

Think about usability. How about making a policy to use Org read-only
buffers for Org agenda, as good example? Expand the Org agenda buffer
with hyperlinks:

- if user press a it is for week or day
                                    ^^^
- but if user clicks on "day" it will show only day

- use hyperlinks to explain more to user or guide one
				    
We already talked that the buffer for org-agenda is really way to
rigid and asks for closing the buffer, opening it, closing, until user
finds its way.

> > ... having pretty ugly and not meaningfull unique ID always shown
> > on screen and more or less to a degree disturbing,
> 
> org-custom-properties ;)

Sure and I use it, like :ASSIGNED: and I also use database IDs. It is
a lot of work for many Org nodes to be handled like that. Each time
assign, insert custom ID, invoke this, that. No, I do not do that. I
count (about) the number of keystrokes and repetitive actions by
observing what I am doing and why and then design faster workflow.

Often I have to export an Org node to ASCII or Unicode, so I have it
as Emasc bookmark.

{C-x r b stag TAB RET C-c C-e C-s t A C-x h M-w C-x k RET C-x b mutt-pro TAB RET C-e RET RET C-y}

That above series of commands may be automatically invoked by using
GNU Hyperbole and action key {M-RET}

GNU Hyperbole
https://www.gnu.org/s/hyperbole

What I do, I jump into Org file, to specific heading, export to plain
text, copy and move to email where I insert it for the person to
receive it.

The workflow I prefer is just:

{s-t stag RET}

I find Emacs {C-x r b} to pull a bookmark also tedious. Function such
as `org-insert-heading-from-pool-as-ascii' where heading is quickly
located by using completion would be useful.

> > ... instead of having
> > them easily editable and changeable and error prone,
> 
> Agree. This is the price of keeping everything in plain text.

I keep everything in plain text too. I just like relating whatever
chunks of plain text to other chunks of plain text. Org does need more
relational approach by any means. It has IDs, custom IDs, properties,
tags, those are all relational attributes. It provides hyperlinks to
related objects. Org is about relations. What is currently lacking is
what partially has been developed and that is integration with other
objects within Emacs or maybe outside, like automated ID generation,
heading wizards, automated hyperlink generation, back links and so on.

Heading wizard could ask user:

- is it related to person?
- organization?
- another task?
- is it assigned to some user?

then after few questions the new Org heading could be created pulling
all those relations into the heading.

** TODO Wash all cloths collected on the first floor
   [Organization name] as hyperlink
   [Manager name]: [Manager phone]
   [Responsible for floor]: [Name]
   Related to task: [3232]

> > ... instead of many
> > actions related to that unique ID,
> 
> Sorry, I do not understand which actions you refer to.

Let us say I am in heading related to organization ABC, there are many
actions connected like calling, contacting people there, reviewing
list of their people, reviewing their emails.

Maybe I am looking on the heading

** Organization ABC

   [IN THIS SPACE]

in that space could be profile of organization. I click [tasks] and
Org file gets nicely ordered. Or maybe [Agenda] and I get only what
relates to this Organization be it in this file or other file, I click
[People] and get people belonging in this organization, [Call] [SMS]
[Send email] [Web] [Schedule meeting], all hyperlinks there without me
thinking.

> I refer to such features as integration. 
> > - last USER who ever modified the heading, note, task, URL, anything
> >   related to that ID, I can find that last user. Hyperlink could be
> >   automatically generated button that points to on the fly generated
> >   Org buffer showing me such modification.
> 
> I do not understand how you determine the user editing something
> in the database. Or do I misunderstand?

Database like PostgreSQL is accessed by using username,
password. Database know that query is conducted by use "Joe Doe" and
by decision of database designer that may remain as record in some of
database tables.

In my opinion any hypertext system should be multi-user based by its
design. Org mode is customarily single user based. With version
control systems it could be multi user but it did not go to that
direction.

It could be concurrently or simultaneously edited as well. 

> > - There is DATE in the hlinks table, so it is possible to have date as
> >   hyperlink button from where I can click to find other same date
> >   related entries, or I could invoke list to find nearby dates.
> 
> FYI. Clicking on timestamps in org automatically invokes agenda view
> listing all the headings containing timestamps from the same day.

Well what I would not know without you I have hundreds of timestamps
and years of using it and I never clicked on it. And now I speak how I
wish to have more hyperlinks. No, I was not aware really those are
hyperlinks. I never clicked on them. Misunderstanding. While they do
look as hyperlinks I was not thinking they are.

> >  That is my button. I never create those hyperlinks again. This
> >  relates only to the tags, but hyperlinks can be for any other type
> >  of relation.
> 
> FYI. When you have multiple tags in your heading, clicking on the tags
> will automatically show agenda view listing all the headings with
> matching tags.

Thank you. Definitely extremely useful feature!

> > - the heading of meta Org can have a status defined by user anyhow and
> >   I do not mean TODO/DONE but could be anything that changes later
> >   actions of the user, or automatically generates hyperlinks.
> 
> The keywords in org are not limited to TODO/DONE. One can create
> arbitrary set of keywords (representing the status) and even define them
> locally in buffer (see "5.2.5 Setting up keywords for individual files"
> in the manual).

Yes, I am aware of that and use it since years. One group is meant as
done, other as not done, and users can specify it. I am aware of
similarities and that is why I said "I do not mean TODO/DONE" but I
did not mean only status of action. Even keywords like that in Org
mode do not need necessarily to mean status of action for the user,
even though they do mean as when "DONE" is chosen then there is date
when it is "CLOSED".

When types are defined in the database it becomes trivial extending
features, instead of type relating to task such as task being closed
or not closed, pending, etc.

** DONE My task
   CLOSED: [2020-11-29 Sun 14:14]

Example of a different type could relate to article such as REVISION

** REVISED My article
   VERSION: [2020-11-29 Sun 14:14]:[1.3]:[1.2]

Where 1.2 could be hyperlink to previous revision, while 1.3 would be
present revision. But keywords like TODO/DONE and others are designed
for action type only. Version control on file system is related to
whole file, but Org is like a book and it can contain various
structured information, project planning that may be revised and where
users wish to see previous versions.

Because of the Org structure that is hard coded one cannot just
invent, one has to make complex enhancements in Emacs Lisp to again
hard code the type.

> > - possible EXPIRATION entry is there, it could be automatically
> >   hyperlinked to other expired items
> 
> Generally, any database field can be represented in org using property
> drawers (note that you can even set global property drawer at the top of
> the org file before the first heading). For the expiration specifically,
> there is even a package in org - org-expity ;)

Yes, and I use it now rather in reverse. Not that I create it by hand,
I let database create it. Then somebody could participate and edit Org
file, send it back or upload for merging or update.

On export to Org those database fields I can attach to headings, tags
I can attach to headings and similar. But not that I wish to work by
hand and edit it and put all those relations manually. So Org becomes
derivation of meta level organization. I am using the headings for
global property drawer, scarcely, like list of people to which tasks
could be assigned.

If a hyperdocument has some attribute related such as person's name,
then on export there can be property with the hyperlink to that
person, from where one can see profile of the person, SMS, notes,
tasks, people introduced by that person, etc. All those become
hyperlinks just very similar to web based applications where HTML
isautomatically generated with many hyperlinks to related objects. We
work in Emacs so we do not work with HTML and there is more
extensibility allowed then just by doing it with HTML.

There are hundreds of thousands of people in my database, thousands
are used in hyperlinks. Imagine if I need to go into their
information, then copy phone number, construct hyperlink to SMS, copy
person who introduced another person, paste into Org file, construct
link, format and align it, test and verify, go into list of notes,
construct link to notes, insert it. That deserves automation.

> Some things may not be very clear for an arbitrary reader though. For
> example, it took me quite a lot of time to wrap my head around the idea
> of using databases - I was thinking too much in terms of org-mode files.
> Would you consider making a blog post showing your workflows in more
> consistent way with examples and possibly gif demonstrations?

What I see is that you and other people are already using database
that is managed by Org. The more properties, tags, timestamps and
other relations are used the more relational it becomes. You would not
have hard time to understand extensing Org with meta data in the
database. Such extension would liberate Org file from meta data and
make meta data reliable and redundable.

Yes, I will have features posted as videos. Then it will come to GNU
ELPA.
 
> I personally approach this by keeping my files as attachments to org
> headings. Then, the files can be references simply by referencing the
> relevant heading (equivalent of your record in the database).

That is good approach. It makes references static and the reference
modifiable. Such solutions help when we come to complexities. If we
have simpler information they may not be necessary.

> Have you ever been in scenario when you want to file something to
> multiple locations?

Many times. One can use hard links or symbolic links on a file
system. Better is using it by relations and defining groups. If group
of sciences consists of biology and physics, maybe photosintesis
belongs to both of them.

If one has to file same thing in multiple Org files on a meta level
there is easy solution. Just creating a new node that points to other
node. Action on a hyperdocument that means "other hyperdocument" would
be the same as on that other hyperdocument. But referencing
hyperdocument would just have ID identifier not a content inside if
that content should be centrally editable.

I have table emails and table pages. If I wish to send page by email I
just tell into the row for the email that the email with ID 3 has the
body text that is actually the page body from page ID 234. When email
is sent, basically page body is sent.

When I define a page I can also say "this page is other page" and this
page acts as other page.

If I define a group of people like Joe, James, Jane, with the ID 4
then I could choose to file under ID 4 as that belongs to group of
those 3 people. Accessing any of those people individually would give
me listing of files filed to the group where person belong and
hyperlinks to those other persons.

Better is when filing of anything has its own ID but is related to the
group as then the filing or document has its own title, annotation and
maybe set of other hyperlinks to hyperdocuments, not only files.

Right now is for me trivial to file into multiple categories when
using database as meta level Org file preparation. For Org file itself
is maybe not trivial.

I can display the list of desired sets or search for them, mark the set and it gets remembered in the list like:

Marked ID 34739 ((34739))
Marked ID 34740 ((34740 34739))
Marked ID 34741 ((34741 34740 34739))

Then what I do with those ID numbers is up to me. I can say update TAG
for each of them, modify their timestamps, close them all at once,
schedule them together, generate group of people, delete them if
necessary, send them by email, anything imaginable. I have already
many such functions and they are simple as they deal with IDs and not
parsing of text.

This is function that marks multiple contacts and creates a group for
those contacts. Because groups are generic they could be groups of
anything like groups of pages, tasks, quick tasks, hyperdocuments,
invoices, opportunities, variables, units, currencies, statistics,
SMS, emails, payment methods, videos, etc.

(defun hyperscope-marked-contacts-to-group ()
  (interactive)
  (let ((id (tabulated-list-get-id)))
    (when id
      (let ((group (rcd-db-get-entry "hlinks" "hlinks_groups" id *hs*)))
	(when group
	  (let* ((type (rcd-db-get-entry "groups" "groups_grouptypes" group *hs*))
		 (table (rcd-db-get-entry "grouptypes" "grouptypes_table" type *hs*)))
	    (when (string= "contacts" table)
	      (let ((array (rcd-sql-int-list-to-array hyperscope-marked-items))
		    (sql (format "UPDATE groups SET groups_array = %s WHERE groups_id = %s" array group)))
		(rcd-sql sql *hs*)))))))))

That would work this way:

- make a group name
  
- filter contacts to get let us say Joe, James, Jane but it could be in multiple accesses.

- press `m' to mark contacts_id 

- invoke function to update contacts for that group

In this case it is group that is related to hyperdocument on which I
am located within tabulated-list-mode, it would be updating group of
people to which this note would be related.

Org way of doing things is to parse everything together each time a
new which essentially is same what database is doing only blazing fast
and error prone. There can be many Joes, James, etc. It is
distinguished clearly due to unique IDs and other attributes.

Now when you have a group defined in a similar speedy manner like
marking files in Dired you can file then anything to that group be it
file, WWW, note, TODO, action, just anything. Additionally
hyperdocument can be related to various other objects such as contact,
person that this contact is assigned to, even group that it is
assigned to. Hyperdocument such as TASK can be related to group of 3
people but assigned to group of other 4 people who need to handle the
deal. In the same time TASK can belong to company ABC, under business
XYZ, as company ABC has multiple businesses. And it could represent a
note to business opportunity X, that belongs in planning and project
Y.

When editing such relations in Org file I would be repetitively spend
time writing, typing, aligning text, formatting, invoking key bindings
and so on. But such editing is not plain text editing any more and we
manage Org and speak of plain text when it is not any more, it
requires editing of structured data. The more data there is the more
complicated it becomes.

Would Emacs have GDBM database built-in I am very sure that Org would
be using the database from its inception and many other packages as
Emacs is already now using various files as replacement for databases
such as bookmarks, saved history, desktop and similar. It needs
external memory that persist even when it is not running. Interesting
it that such can be accessed by any programming language than.

> > c a - I am editing author's name in free text. If author is in the
> >       database I would choose author. Author's name can be hyperlinked to
> >       find other entries under same author. No need to type it by hand.
> 
> Do you need to set the link to author every single time you create a new
> entry authored/related by/to that author? Or do you somehow create the
> link to the author automatically?

I have 2 columns, one is hlink_author related to table `contacts'
having priority and other is hlink_authorname which may be used when
author is not in the contacts table.

If there is hyperdocument "Book ABC" and I need to enter Book XYZ
under same node, I would simple `C' to duplicate the node which
duplicates all meta data and I just curate the name of the book and
actual hyperlink (URI) to the book. This duplicates author.

Otherway is to filter various items belonging to author, mark those
items and then update author for all of them at once.

Once value is updated, the description of the hyperdocument is shown
similar like this below and I can click on Doug Engelbart to get all
entries related to author. But I do not create the hyperlink to author
any more. Anything that is already assigned as relation should have
hyperlinks automatically to help in faster access to information.

Doug Engelbart's Design
=======================
   Author: Doug Engelbart
     Type: Set ➜
Hyperlink: Doug Engelbart's Design

> > T - I am editing tags freely, I can enter: emacs lisp todo review; and
> >     those words would become tags. After ENTER I can immediately see
> >     hyperlinks to those tags.
> 
> Are your tags also uniquely defined using relational database? Say, you
> want to change some tag name for some reason. Will changing the name
> automatically update all the tags in the entries marked with that tag?

Good question. Answer is yes and not yet as I have `tags' table since
years and anything can be inside.

                                    Table "public.tags"
      Column      |  Type   | Collation | Nullable |                Default                
------------------+---------+-----------+----------+---------------------------------------
 tags_id          | integer |           | not null | nextval('tags_tags_id_seq'::regclass)
 tags_name        | text    |           | not null | 
 tags_description | text    |           |          | 

 But hyperdocuments are now tagged freely as a string with spaces as separator:

 emacs-lisp gnu lisp

 would be 3 tags. Upper case does not matter.

 I could as well enter the loop where I choose multiple tags from the
 table `tags' and relate it to the table. What I wish to say is that
 when one works with database tables, unique IDs, things become very
 simple and many checks, verifications, and data or variable handlings
 become redundant.

> > If the task is assigned to Communication Officer or senior who has to
> > advise Joe how to do it, then on the next key press I could get
> > various hyperlinks because there is CID-1 that relates to Joe Doe:
> > 
> > * Joe to purchase ticket in Kahama and travel on Saturday to Dar es Salaam  :CID-1:
> > 
> >   [Joe's files] [Initiate call] [SMS to Joe] [Send this task to Joe] [Plain email to Joe]
> 
> This is an interesting idea. You mentioned it in the past, but this
> example made it much more clear.
> 
> Something similar is implemented in org-ref
> (https://github.com/jkitchin/org-ref). Instead of adding links like
> [Joe's files] directly into the text, org-ref shows various
> possibilities when one clicks on the link itself - there is a completion
> dialogue suggesting actions related to the link.

That is great. I have seen pictures and that sounds how it should be
to help user obtain references and do various actions with them. It
also again shows the need for more structured and more relational
approach. Only due to experience with SQL and easy of organizing I
have the viewpoint that Org developers work much and people put much
efforts in construction of something that is generally and have been
constructed before with database approaches. Without experience with
PostgreSQL I would not know it.

The more and more enhancements are there for Org, but I do not get
excited really. To people looking on it, it looks great. I am not
excited about that.

org-reg demonstrates good integration to bibtex entries.. It provides
necessary relations of list of objects (table in database) to text and
other objects. It creates relations and for that reason provides
better integration and use.

The BibTeX entries belong to TeX and LaTeX. There are also Emacs
packages for editing such.
https://en.wikipedia.org/wiki/BibTeX#Entry_types

My table hlinks (hyperlinks) contains one column being array:

hlinks_array text[]

which means that such node can be defined as BibTeX type and that such
type has entry types stored in the array just as described:

Article: author, title, journal, year, volume
Book: author, editor, publisher, year, title
etc.

It could in itself as whole table be defined to be BibTeX compatible
or it could just use the table column hlinks_array which would hold the BibTeX entry:

hlinks_type could be "BibTeX Article" with the title being TITLE,
AUTHOR being referenced to table `contacts' or `authors', JOURNAL
field could also reference bunch of journals, YEAR can be entered
freely and volume, etc.

BibTeX is prime example where database editing gets its use. No need
to repetition as if author has been entered once then such author may
be chosen in other 23 entries by using just few letters to complete
author's name and find the unique ID. In fact hlinks table is somewhat
similar but does not complete BibTeX specification. It could complete
if I work 30 minutes on it.

> Since org-ref is specifically designed for referencing scientific
> articles, it implements only a single set of suggestions like: (1)
> open pdf of the article; (2) show notes; (3) email article to
> someone; (4) open url page of the article, ...

That is good observation. If org-ref would be designed on meta level
it would not hold only BibTeX, it could hold anything and become a new
Memex similar to HyperScope becoming Memex in its definition. For now
it only helps in providing references by providing simple relation to
BibTeX entries. But it does not fixes those relations.

> > Because specific paragraph referencing is now not easily possible I am
> > using database approach of editing chunks of the Org file where then
> > any piece of Org can become referential.
> 
> It is actually possible to reference arbitrary chunks of text in org:
> 
> "4.2 Internal Links" section of the manual:
> 
>      1. one item
>      2. <<target>>another item
>      Here we refer to item [[target]].

Somebody referred that to me, maybe you, before few weeks as I wanted
to export Org file that I am producing on the fly with fine
referencing. That is where I realized that it is not built in for
paragraphs or items.

** Exported 
1. one item
2. <<target>>another item
Here we refer to item [[target]].

Exported would look like this:

1. one item
2. another item
Here we refer to item 2.

So it loses its references in export. Not that I find it amusing. I
would like to have unique references in export and also shown to
people.

I was considering to go over all the nodes or headings and to inject
<<target>> for every paragraph to get a finely referenced
document. But that is again hackish and complex and makes documents
ugly. So I have decided to transition into multi-object structures.

That approach for me to parse text and inject targets does not comply
to:

TECHNOLOGY TEMPLATE PROJECT OHS Framework 
https://www.dougengelbart.org/content/view/110/460/

See:

Hyperdocument Content and Structure [2a]
https://www.dougengelbart.org/content/view/110/460/#2a

You may also observe how Doug Engelbart's Institute website references
hyperdocuments finely grained by using [2a] reference in this
case. When teaching people in the field of mining I have need to
reference specifics, otherwise people get confused.

,----
|  Elementary Objects [2a1a]
| 
| Objects are basic content packets of an arbitrary, user and developer
| extensible nature. Types of elementary objects could contain text,
| graphics, equations, tables, spreadsheets, canned-images, video,
| sound, code elements, etc.
`----

That is what I have created. It is very simple:

1. Create table for types that can be user and developer extensible
2. Create table for elementary objects. In my case called hlinks

Finished there. If type is spreadsheet, image, video, etc. it is
designated upon creation of elementary object.

Org file is going in reverse. Org has not been defined that way from
ground up, but developers try to fit headings into elementary objects,
but it does not work well.

,----
| Mixed-Object Documents [2a1b]
| 
| Documents are a coherent entity made up of an arbitrary mix of
| elementary objects bundled within a common "envelope" to be stored,
| transmitted, read, printed, or otherwise be operated on.  2a1b1
| 
| the MIME specification is an example of a document definition based on
| a higher level collection of elementary objects.  2a1b2
`----

On my side any node can be also hyperdocument. But if I wish to have
hyperdocument consisting of many other mixed objects that is what I
have defined as a "Set" which is Mixed-Object Document. I can export
mixed-object document into Org file as example. I could as well export
it as directory, or export it as tar package or full HTML page that
contains all those objects hyperlinked to each other, or actually
construct MIME message and send everything together to somebody.

,----
| Objects 
| 
| Objects and the documents made out of them are shareable among members
| of a networked community: they may be simultaneously viewable,
| manipulable, and changeable according to rules appropriate to their
| nature and supported in access permissions and restrictions contained
| in their definitions.  2a1c1
`----

Hyperscope is PostgreSQL database backed so multi users can access one
database or use software to access multiple databases or copy from one
to each other various objects or reference from one database to other
those objects, or export objects into file system or into some type of
files like Org or HTML where objects can be again re-used , shared,
viewed, edited.

Database has concurrency support, all users can edit it in the same
time with version control. Access permission and restrictions are
defined on the database level and on the software level. For example
if user has access to modify some value I have a version control on
the software level (for now) but user could skip the version control
and modify value without having previous version stored in the
database.

,----
| Object ID-Time
| 
| Each creation or modification of an object automatically results in
| the creation of a stamp containing information concerning the date,
| time, date, and user identification associated with that modification.
| 2a1d1
| 
| Users can filter and control the portrayal of this information and
| their associated content objects
`----

This requirement becomes trivial. When creating hyperdocument `hlink'
entry there is automated entry date-created, date-modified,
user-created, user-modified and I have now version control where
history of modifications can be stored as well.

This becomes important in multi user environment, as then agenda gets
additional meanings:

- tasks modified by Joe since February 20th

- tasks entered by Jane, modified by James

- tasks never modified in past

,----
| Personal Signature Encryption/Verification/Authentication of Objects and Documents
| 
| Users can affix personal signatures to a document, or a specified
| collections of objects within the document, using private signature
| keys. Users can verify that the signature is authentic and that no bit
| of the signed document or document segment has been altered since it
| was signed. Signed document segments can be copied or moved in full
| without interfering with later signature verification.  2a1e1
`----

I keep: `hlinks_signature' table entry for this. Now I see that I have
to add signature column to version control table.

,----
| ALTER TABLE vc ADD COLUMN vc_signature TEXT;
| ALTER TABLE
`----

Done.

This would mean that version table could optionally receive GPG/PGP
signature of the previous version of some single value. Not the whole
document in this case. But actual document could contain the
signature. If it is file, the file could be signed. If it is "Set"
then name and ID of the set could be signed. If it is note, the note
in memory can be signed. If it is task, then it can be signed. This is
important feature especially in remote transactions.

Not one time it happened that staff member tells other to disburse
money when it is not ordered. If there is rule that money cannot be
disbursed unless digitally signed and verified than incidents may be
prevented.

,----
| https://www.dougengelbart.org/content/view/110/460/#2b1a
| 
| Global, Human-Understandable, Object Addresses
| 
| Every object that someone might validly want/need to cite or otherwise
| access should have an unambiguous address, capable of being portrayed
| in a human readable and interpretable manner. Most common existing
| spreadsheet programs have provisions similar to this for cell
| addressibility.  2b1a1
`----

Now we come to <<targets>> as I can place them but they are only meant
for HTML export. I do not get targets in ASCII export. But I do need
it as they are useful to locate the specific section that has to be
found. They could be wrapped in something [[123]] to be easily found
by searching [[123 for example.

If I wish to provide specific target for everything in Org file it
would not be readable any more, rather ugly. Paragraph may change or
be split in two paragraphs, parsers would provide new <<target>> not
being unambiguous.

If preparing structure on meta level, specific paragraphs could be
split but their reference and meaning would remain under one unique
ID.

Let us say somebody wish to write a document and chunk it into
referencable paragraphs. One could write one node as a document of 4
pages. Once document has been finished, finalized, then it could be
automatically chunked into paragraphs each having its reference.

,----
| Linking/Filtering/Access Control
| 
| The Basic "Hyper" Characteristics
| 
| Embedded objects called links can point to any arbitrary object within
| the document, or within another document in a specified domain of
| documents. Links can be actuated by a user or an automatic process to
| "go see what is at the other end," or "bring the other-end object to
| this location," or "execute the process identified at the other end."
| These executable processes may control peripheral devices such as CD
| ROM, video-disk players, etc.  2b2a1
`----

- can I point to any other object within the document? I can if I use
  multi-object document.

  I could also construct link that points to specific lines or
  searches within same document similar like:
  {M-x highlight-lines-matching-regexp RET hyper RET hi-yellow RET}

  Who is using GNU Hyperbole would know be able to press{M-RET} on the
  above hyperlink within {M-x ...} and would see the lines highlighted
  in yellow.

  Other systems within Emacs may be used to point out to same parts of
  the text within one specific document.

,----
| Hyperdocument "Back-Links"
| 
| Information about other objects pointing to a specific object in a document is available.
| 2b2b1
`----

As long as hyperlinks are in my database or other database that is
accessible to me, a search in the database could locate hyperlinks to
specific objects and find their IDs and provide list of
back-links. Otherwise without search and without reporting that
somebody is pointing to it, I do not know how would I implement this
feature.

Other objects pointing to one hyperdocument signify there is some
relation involved and are necessary to be discovered.

,----
| Access Control
| 
| Hyperdocuments in personal, group, and library files can have access
| restrictions down to the object level based on individual identity or
| organizational role.  2b2c1
| 
| Access controls can include, for example, allowing or disallowing a
| range of access types such as read, write, or knowledge about
| existence.  2b2c2
`----

PostgreSQL is great as it does allow row security policies:
https://www.postgresql.org/docs/13/ddl-rowsecurity.html

My first surprise question when I discovered Org mode was:

,----
|   why I can assign or delegate a task but cannot share it to those
|   people to which I have delegated it?
`----

  This question becomes more surprising when one can see mobile
  devices have solved the problem of sharing and integrated it very
  well while desktop and notebook computers did not. It should be
  built-in feature if we speak of tasks.

> Or using #+NAME: keyword
> 
>      #+NAME: My Target
>      | a  | table      |
>      |----+------------|
>      | of | four cells |
> 
> Unfortunately, it only works inside a single file. Though I imagine that
> org-id can be modified to support global referencing.
> 
> You can send a feature request about this if you are interested.

In general I am satisfied with Org. Our brainstorming could be used by
those who may read it to enhance it how they wish. My comments are
there for thinking on what could be useful.

Recently I found very useful that `browse-url-handlers' can be
customized so that Emacs can open any kind of hyperlinks as long as
`goto-address-mode' is turned on:

gopher://example.com appears as hyperlink

mid:mailbox:message-ID would appear as hyperlink (I do not know if this
is proper usage)

gemini://example.com can be defined to appear as hyperlink

This makes things very simple and allows my hyperdocuments to be
opened without using Org mode. For example comments in Emacs Lisp can
have plethora of hyperlinks without Org mode and I do not say
necessary WWW hyperlinks. It could be message ID hyperlinks which
explain better why some function has been created.

By using GNU Hyperbole Emacs Package, one can use M-RET to active this hyperlink:
{C-h v thing-at-point-uri-schemes RET} to see the URI schemes that
Emacs can support.

Its value is

("aaa://" "about:" "acap://" "apt:" "bzr://" "bzr+ssh://"
"attachment:/" "chrome://" "cid:" "content://" "crid://" "cvs://"
"data:" "dav:" "dict://" "doi:" "dns:" "dtn:" "feed:" "file:/"
"finger://" "fish://" "ftp://" "geo:" "git://" "go:" "gopher://"
"h323:" "http://" "https://" "im:" "imap://" "info:" "ipp:" "irc://"
"irc6://" "ircs://" "iris.beep:" "jar:" "ldap://" "ldaps://" "magnet:"
"mailto:" "mid:" "mtqp://" "mupdate://" "news:" "nfs://" "nntp://"
"opaquelocktoken:" "pop://" "pres:" "resource://" "rmi://" "rsync://"
"rtsp://" "rtspu://" "service:" "sftp://" "sip:" "sips:" "smb://"
"sms:" "snmp://" "soap.beep://" "soap.beeps://" "ssh://" "svn://"
"svn+ssh://" "tag:" "tel:" "telnet://" "tftp://" "tip://" "tn3270://"
"udp://" "urn:" "uuid:" "vemmi://" "webcal://" "xri://"
"xmlrpc.beep://" "xmlrpc.beeps://" "z39.50r://" "z39.50s://" "xmpp:"
"fax:" "man:" "mms://" "mmsh://" "modem:" "prospero:" "snews:"
"wais://")

And one could define one owns URIs. Isn't it great? Then it works in
every buffer and every mode with `goto-address-mode' as minor mode.

Write text, on the end of text you may include glossary to words.

Glossary
========
Emacs: dict://dict.org&emacs

Or give instructions which packages to install:

apt:install:emacs (do not know if usage is proper but handler can be
made to make it anyhow)

or embed picture into buffer:

data:base64ababbac (huge)

or access VPS servers one by one to maintain them:

fish://user@example.com

or download map of user's location:

geo:1.2,1.1 (wrong usage here, just demonstration)

or contact person

jabber:louis@gnusocial.club

or

xmpp:louis@gnusocial.club

or call or SMS person:

tel:+1234567890

sms:+1234567890

or open manual:

man:bash and this hyperlink actually work without me setting up handler.

> > Heading A can be one node, edited by user one time. It need not be
> > duplicated for other file, rather only a pointer to Heading A can be
> > included and the Heading then appears in a new Org file.
> 
> That would be a great feature indeed. This idea is actually getting a
> lot of attention recently:
> https://github.com/alphapapa/transclusion-in-emacs

Those are logical user demands that come from their practice. When
using database that becomes trivial:

1. Assign the source node to be of type "Org heading" with its "body".
  
2. Assign the symbolical node to be symbolical to point to the node 1.

If I try to implement the centralized database like approach, then
this could be workflow:

1. Create node
   
2. Refile it as copy to specific Org file for source nodes. Such could
   be designated in the form of a hyperlink.
   
3. Assign property to symbolic nodes in the other Org file that it
   becomes clear they are symbolic to the original. Symbolic node can
   be anything but it would convert itself into the source node.

4. By using key the symbolic node could inject itself into the
   original.

** Source node
   :PROPERTIES:
   :TEE-ID:   1
   :END:

Some text

** Symbolic node, call it as you wish as it would replace itself with source node
   :PROPERTIES:
   :SYMBOLIC-TO: ~/Documents/Org/Original-File.org::1
   :END:

   Anything here.

** For function

(defun org-fetch-insert-tee-heading ()
  "Replaces Org node with SYMBOLIC-TO property pointing to
FILE::ID with the Org node from FILE and property TEE-ID being ID"
  (interactive)
  (let* ((symbolic-to (org-property-values "SYMBOLIC-TO"))
	(source (split-string (car symbolic-to) "::"))
	(source-file (car source))
	(source-id (cadr source)))
    (when (and source-file source-id)
      (let* ((heading (with-temp-buffer
			(insert-file source-file)
			(org-mode)
			(goto-char (org-find-property "TEE-ID" source-id))
			(org-get-entry)
			(buffer-string)))
	     (replacement (with-temp-buffer
			    (insert heading)
			    (org-mode)
			    (goto-char (org-find-property "TEE-ID" source-id))
			    (org-delete-property "TEE-ID")
			    (org-set-property "SYMBOLIC-TO" (concat source-file "::" source-id))
			    (buffer-string))))
	(org-get-heading)
	(org-cut-special)
	(insert replacement)))))

Function works and it is concept. It can or should be improved.

Workflow is:

1. Designate property in a file with TEE-ID (remind me of `tee'
   shell command) and assign the ID as you wish.

2. Provide property in other nodes with :SYMBOLIC-TO: to point to file
   and the ID.

3. Assign function to find all symbolical nodes to replace itself with
   the original node.

4. Another function could help to "upload" the source node with
   whatever has been edited in the in symbolical nodes.

Re: Is Org really so simple?

[Dr. Arne Babenhauserheide]

[-- Attachment #1: Type: text/plain, Size: 1488 bytes --]


Ihor Radchenko <yantar92@gmail.com> writes:

>> Only philosophy I know is that it is plain text. Is there any official
>> philosophy? I have no idea, at least manual does not give me
>> references. I cannot find "philosophy", send me references.
>
> You are right. There is no official "philosophy" in org. In my reply I
> tried to follow the topic starter's view:
>
>  Texas Cyberthal <texas.cyberthal@gmail.com>:
>> By philosophy, I mean the dev consensus on the correct way to do
>> things, and coded configuration and usability biases.
>
> According to my experience with org-mode development (I am not talking
> about third-party packages here), it is discouraged to change org-mode
> towards hiding metadata in org files or store *unique* data (that cannot
> be derived from the contents of the org files) related to org-mode
> externally (not in org files). It is not official statement, but rather
> my impression so far.

As a user I do not want outside-data, because I put my org-files into
versiontracking and access them from several machines. I also provide
them as sources for books on sr.ht and use them to synchronize planning
between the office-PC and the homeoffice-PC. And I send them to people
so they have the complete source of something I built.

All those use-cases I use regularly would break if org-mode started to
rely on an external database.

Best wishes,
Arne
-- 
Unpolitisch sein
heißt politisch sein
ohne es zu merken

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 1125 bytes --]

Re: Is Org really so simple?

[David Rogers]

Jean Louis <bugs@gnu.support> writes:

> * Ihor Radchenko <yantar92@gmail.com> [2020-11-23 17:18]:
>   :PROPERTIES:
>   :CREATED:  [2020-11-23 Mon 18:42]
>   :ID:       edebb3e7-e755-4ecc-a5e8-a3353a3f5fd0
>   :END:
>> Dear Jean Louis,
>> 
>> Your description of the database reminds me how org-roam 
>> handles the
>> files - it also uses an external database for linking and 
>> allows quick
>> incremental search that does not really depend on where the
>> files/headings are stored.
>
> Sounds good, I can see there is graph database used.
>
>> However, what you are talking about is against org-mode 
>> philosophy,
>> as I know it.
>
> Only philosophy I know is that it is plain text. Is there any 
> official
> philosophy? I have no idea, at least manual does not give me
> references. I cannot find "philosophy", send me references.
>
> It says "to keep simple things simple". But Org is far far from 
> being
> simple any more. It offers good principles, paradigms and people 
> built
> many enhancements upon those. Speedily it becomes way much more 
> than
> simple.

Nothing was mentioned about keeping Org-mode simple. You’ve made a 
bad misreading there. It said keeping *simple things* simple - in 
other words, avoid taking a simple thing and making it 
complicated. Things that really are complicated (“in real life”) 
may *sometimes* be simplified, and that might be good - but “Make 
everything be simple” is not a valid goal for any useful piece of 
software. Often, a complicated thing must stay complicated.

In my opinion, this kind of “simple vs complex” discussion is like 
discussing the quality of competitive sports teams. It’s simple at 
one level - which team has the best players? But in reality, 
there’s a lot more to it: Is there enough money? Do these players 
work well with this coach? etc. In other words, support and 
infrastructure. If you focus only on the simple part, it becomes 
like taking that good group of players and putting them on the 
moon by themselves. In reality, they can only be a good team when 
all of the complicated messy infrastructure is functioning, and 
without the infrastructure, they don’t function correctly.

Org-mode is a clever way of “leveraging” Emacs. Org-mode isn’t a 
self-contained application - instead, it’s more like a (large) 
Emacs plug-in.




Or maybe this:

Copying the design of a hydroelectric dam and generator, and 
building an exact duplicate in the middle of a desert, isn’t 
effective. The concept is simple (the turbine spins, it generates 
electricity) - but for it to work, you have to have the right 
thing to build it on top of. Rivers are harder to build than 
hydroelectric generators are. :) You don’t carry around a 
hydroelectric dam looking for a place to put it - you only start 
building after you’ve found a good river, and you build 
differently depending on the circumstances.

-- 
David

Re: Is Org really so simple?

[Tim Cross]

David Rogers <davidandrewrogers@gmail.com> writes:

> Jean Louis <bugs@gnu.support> writes:
>
>> * Ihor Radchenko <yantar92@gmail.com> [2020-11-23 17:18]:
>>   :PROPERTIES:
>>   :CREATED:  [2020-11-23 Mon 18:42]
>>   :ID:       edebb3e7-e755-4ecc-a5e8-a3353a3f5fd0
>>   :END:
>>> Dear Jean Louis,
>>>
>>> Your description of the database reminds me how org-roam
>>> handles the
>>> files - it also uses an external database for linking and
>>> allows quick
>>> incremental search that does not really depend on where the
>>> files/headings are stored.
>>
>> Sounds good, I can see there is graph database used.
>>
>>> However, what you are talking about is against org-mode
>>> philosophy,
>>> as I know it.
>>
>> Only philosophy I know is that it is plain text. Is there any
>> official
>> philosophy? I have no idea, at least manual does not give me
>> references. I cannot find "philosophy", send me references.
>>
>> It says "to keep simple things simple". But Org is far far from
>> being
>> simple any more. It offers good principles, paradigms and people
>> built
>> many enhancements upon those. Speedily it becomes way much more
>> than
>> simple.
>
> Nothing was mentioned about keeping Org-mode simple. You’ve made a
> bad misreading there. It said keeping *simple things* simple - in
> other words, avoid taking a simple thing and making it
> complicated. Things that really are complicated (“in real life”)
> may *sometimes* be simplified, and that might be good - but “Make
> everything be simple” is not a valid goal for any useful piece of
> software. Often, a complicated thing must stay complicated.
>

“Everything should be made as simple as possible, but no simpler.” -
Einstein.

--
Tim Cross

Re: Is Org really so simple?

[Jean Louis]

* David Rogers <davidandrewrogers@gmail.com> [2020-11-27 02:10]:
> Nothing was mentioned about keeping Org-mode simple.

Good point.

> In my opinion, this kind of “simple vs complex” discussion is like
> discussing the quality of competitive sports teams. It’s simple at one level

> Org-mode is a clever way of “leveraging” Emacs. Org-mode isn’t a
> self-contained application - instead, it’s more like a (large) Emacs
> plug-in.

I find it good that Org functionality is available outside of Org mode
and in that sense enhances possibly many other areas of Emacs, such as
org-link-minor-mode that works in any other mode.

Re: One vs many directories

[Texas Cyberthal]

Hi Jean,

> I have tried your solution and could not find the mental concept to relate to my thinking.

I forgot this inductive sorting skill must be learned gradually, like
touch typing, at small scale before exomind conversion.

> Do we think of a tree of knowledge first? I do not think so. And there are memory systems that DO think of plethora of various things and increase human memory capabilities.

Yes, Textmind becomes a mnemonic system.  The tree associates all
one's info together, making explicit one's personal implicit
prioritization of info.  Doing so systematically is only possible with
computer plus user algorithm.

> How human think -- is nowhere defined and is vague. Human thinks how they think and there may be as many versions as humans.

The brain is plastic.  It adapts easily to sync with a Textmind tree.
This tree's complete thought algorithm is an improvement over native
thought pattern.  Computer and brain meet in the middle.  That is
cognitive cyborg first stage.  Keyboard+screen is Brain-Computer
Interface.

The other complete thought algorithm is Pubmind, for longform content.
But it doesn't work without Textmind.

Other thought methods are even less complete, and thus more dependent
on the Textmind foundation.  For example, pure association and search
retrieval benefits greatly from Textmind de facto spaced repetition
and directory scoping.

> Doug Engelbart has already envisioned how files could be stored, accessed, hyperlinked, referenced and we do not use it in that sense today after how many years?

Exactly.  To the extent he was correct, his ideas have been adopted.
To the extent wrong, ignored.

The main problem is sustaining long-term hybrid-intelligence text-mind
sync.  It requires a complete OODA algorithm.  Engelbart doesn't think
on this scale.  David Allen's GTD tries to, but is limited by paper.

One can always improve the crystallized knowledge of a PIMS by adding
more metadata and links.  That misses the point: fluid intelligence is
more important.  It tells which info is worth promoting to more
expensive representations.

Re: One vs many directories

[Jean Louis]

* Texas Cyberthal <texas.cyberthal@gmail.com> [2020-11-23 19:08]:
> Hi Jean,
> 
> > I have tried your solution and could not find the mental concept to relate to my thinking.
> 
> I forgot this inductive sorting skill must be learned gradually, like
> touch typing, at small scale before exomind conversion.

I find it entertaining for now. Now, what is exomind? 

> > Do we think of a tree of knowledge first? I do not think so. And there are memory systems that DO think of plethora of various things and increase human memory capabilities.
> 
> Yes, Textmind becomes a mnemonic system.  The tree associates all
> one's info together, making explicit one's personal implicit
> prioritization of info.  Doing so systematically is only possible with
> computer plus user algorithm.

I do agree that various systems like 10 Bins for filing files by how
human thinks are useful. Only that "how human thinks" varies by
humans and not even humans may know how to explain how they think.

My concept is that I think what I want, like "I wish to see history of
interactions with this person" and then I click F3 or something, I can
see SMS, emails, if there were any negotiations, contracts, if there
are licenses submitted for partnership. That all has some practical
meanings. As for now and probably due to misunderstanding I cannot
find 10 Bin practical for me and this may be due to different way of
thinking.

Could you give me reference describing how you would file:

- Contract signed between your company 123 and person ABC?

- Image on which there is only your family member, not you, which has
  its date?

- Image of you, with the date?

- Image without date in the file name and not embedded?

- Software git tree related to mailing things?

> > How human think -- is nowhere defined and is vague. Human thinks how they think and there may be as many versions as humans.
> 
> The brain is plastic.  It adapts easily to sync with a Textmind tree.
> This tree's complete thought algorithm is an improvement over native
> thought pattern.  Computer and brain meet in the middle.  That is
> cognitive cyborg first stage.  Keyboard+screen is Brain-Computer
> Interface.

OK while I do try to adapt for now I do not find it easy. While I do
not adore hierarchies, file system is easier to adapt, as user can
just something like "Brother" and file it there.

It is entertaining as it sounds futuristic. But I do not know thought
algorithm, and what would be native thought pattern, and I would not
like to become cognitive cyborg, not in this stage of Emacs
development, as nobody likes to get killed. Maybe in some future when
I can load it into my personal memory and run on it even during sleep
time.

Keyboards will soon expire, they partially already expired as billions
of people use computers without keyboards including that term
"computer" became dilluted and hidden so that people do not even know
they carry one or two in their pockets. Screen will expire
too. Keyboards will become sensitive lights and screens holograms.

> The other complete thought algorithm is Pubmind, for longform content.
> But it doesn't work without Textmind.

Need practical example.

Re: One vs many directories

[Ihor Radchenko]

> I find it entertaining for now. Now, what is exomind? 

Unless I misunderstood, Jean referred to "external brain" concept:
- https://beepb00p.xyz/exobrain/
- https://zettelkasten.de/posts/extend-your-mind-and-memory-with-a-zettelkasten/
- https://github.com/novoid/Memacs
- https://blog.jethro.dev/posts/org_mode_workflow_preview/

Best,
Ihor

Jean Louis <bugs@gnu.support> writes:

> * Texas Cyberthal <texas.cyberthal@gmail.com> [2020-11-23 19:08]:
>> Hi Jean,
>> 
>> > I have tried your solution and could not find the mental concept to relate to my thinking.
>> 
>> I forgot this inductive sorting skill must be learned gradually, like
>> touch typing, at small scale before exomind conversion.
>
> I find it entertaining for now. Now, what is exomind? 

Re: One vs many directories

[Jean Louis]

* Ihor Radchenko <yantar92@gmail.com> [2020-11-24 10:57]:
> > I find it entertaining for now. Now, what is exomind? 
> 
> Unless I misunderstood, Jean referred to "external brain" concept:
> - https://beepb00p.xyz/exobrain/

The more you send me reference more I discover other set of people
doing same what I am doing. Since I have implemented central meta
level organization it is moving rapidly, everthing gets sorted. It
develops by itself and is rapidly accessible.

That website I have to mirror locally to pick ideas and learn from
others. Mirroring I do with:

$ wget -Emk http://example.com

As that command replaces all hyperlinks to local hyperlinks. That
person advanced in organization of things. I stick to few principles
and just design it by principles.

Design works rapidly. Few Emacs Lisp functions and access to reports
listed in Emacs Buffers and integration with other tools.

With one function and one PostgreSQL table defined in 3 minutes I get
rudimentary backup and version system for any column values that I am
editing in the database. If I edit note, the note is versioned
(previous version stored) before I start editing it. Principles I am
following are basics what programmers like, to minimize or eliminate
repetitions and efforts to achieve the goal.

Person above have extracted or exported its own database of hyperlinks
to hyperdocuments. My side I have made for now Org export of any
subtree or the whole dynamic knowledge repository. There are many
things to go. In Emacs development version all kinds of hyperlinks can
get their handlers like gopher:// gemini:// message: tel: sms: and
htat will be very helpful.

No, I do not use "exobrain" as a term. I rather lean on Engelbart's
terminology and follow his principles as we are very late to implement
what was envisioned back in 1968 and before. It is 52 years already.

And many more years since Memex has been invented:

Memex
https://en.wikipedia.org/wiki/Memex

As author said: "The memex device as described by Bush "would use
microfilm storage, dry photography, and analog computing to give
postwar scholars access to a huge, indexed repository of knowledge any
section of which could be called up with a few keystrokes."

And that is exactly what I am creating here to have anything called up
with few keystrokes and to be able to share files with individuals or
groups of people without more thinking but just designated what to be
done.

Have group of 5 people to share notes with? Just find the designated
group and click share. Computer would handle the rest, maybe send
files by emails individually, maybe inform people by SMS, maybe upload
files and share password protected hyperlinks with those people.

Integration is another keyword I like to follow. Android principle of
sharing is pretty much based on integration. We have all the small
functions around us only not well integrated with their relations that
concern human problems.

We have files on file system which we cannot easily share with groups
or people we want. Address books are all sparse, one is in this email
client, one is separate, one is on the mobile device, another email
client does not synchronize, and so on. I have forgotten this long
ago and use central address book from where everything derive:

- no Google, clouds, etc. that is very insecure. Do not give contacts
  to Google, there are hundreds of thousands of staff members there
  and no guarantee whatsoever that they will not read it.

- keeping contacts on my computers. I have already spent money for
  hard disk, there is enough space

- exporting contacts from central database and importing to email
  clients, mobile devices, this way everything is synchronized.

How quickly can GNU/Linux user share a file with somebody?

- locate the file by using hierarchical browsing. If file system is a
  mess, this alone may take some time

- open up email reader

- find that email address. If it is in the email reader already it is
  good. But it could be in the phone. It could be on paper, or on
  business card. Where is it? Maybe calling person? But where is the
  phone number? On first phone, second phone... if all is synchronized
  maybe is easier to find.

- attach the file

- send the file.

But then sending SMS or calling in the same time does not
work. The above process is not well integrated.

It could work like this:

- user just thinks of what has to be shared with other person, types
  the terms related to the thought

- locates the file and press share

- locates the user and press enter. FINISHED

That would be better integration. Even better it would be if user can
choose the automated workflow option:

1. send the file, automatically record that file has been sent to
   specific user. Tell user automatically how many files are attached
   and attach annotation belonging to the file as body of the email or
   any instructions.

2. in the same time inform the user by SMS that file has been sent and
   record that SMS have been sent. Software like kdeconnect, gnokii
   can be used for it.

3. within 1 hour, or other period of time, computer asks to initiate
   the call to the user to follow up about the file sent and maybe
   nudges few times and records the action. Software like termux tools
   can be used for it.

My first big surprise with Org was that there was no possibility to
assign the task to other person and send that task! I actually could
not believe that it was meant for single person or personal tasks and
notes. Then I made the function to share the task quickly to any
person assigned to the task. If person is assigned, task is sent to
the person. If no person is assigned then I choose to which person to
send it. This includes also groups of people.

> - https://zettelkasten.de/posts/extend-your-mind-and-memory-with-a-zettelkasten/

That is similar idea of organizing. There is claim that one shall
forget about categories and rather use tags. I think that using any
types of attributes is better and using more attributes helps in
quicker location.

> - https://github.com/novoid/Memacs

I have installed it and not yet used it. I would not like having too
many tools on file system to manage information. There are too many
memacs tools made for console. In general I am tracking all SMS sent
from phones to other people, they are automatically inserted into
corresponding people's objects. Then I know which people received
what. Phone calls can be tracked too. Phone calls can be
recorded. Sales and marketing departments need that. I am using now
only principles from Memacs and implement some of them in Emacs
Lisp. As I like integration I do not like external tools, but the
dynanic knowledge repository must be usable externally without
Emacs.

That becomes very easy by expanding the whole tree of notes into the
file system from time to time and generating meanings for symlinks
that point to fixed locations on file system.

The centralized subtrees or nodes of my dynamic knowledge repository
can be moved easily from one parent node to other parent node. This is
because human must sort things properly. But if such are pointing to
files on file system those files never change. Meaning and relations
can change but file location should not change. Directories are more
static then files. They would never change. Files if not indexed but
located in the archive could maybe change or get updated.

Git repository could get updated but its directory need never change
in the future.

Let us say there are many PDFs to be indexed and accessed through
semantics. The PDF file name could change but access to PDF file need
never change. Renaming PDF file need not change access to PDF file in
other word there is no need to rename it twice, it can remain in the
database and get accessed automatically. But that requires directory
to be static, or it requires md5sum of the file to be
static. Something must be static that file can be found by the
system. Best is when file is under specific unique ID that never
changes. Then everything becomes unique and clear. And symlink can be
automatically generated:

~/hyperscope/1/2/3/432.pdf would be file

~/hyperscope/1/2/3/Knowledge.pdf would be symlink to 432.pdf
automatically generated and from time to time updated if there were
many changes in the database.

The Org hyperlink to the file could point to:
~/hyperscope/1/2/3/432.pdf because file location is this way static
and will never change.

But the Org hyperlink could as well point to meta level hyperlink
(hyperscope 432) as that would open the file no matter where is the
location.

And if file is on remote server, something like

[[PDF File][(hyperscope 432 2)]]

would then work quite well. As this type of dynamic knowledge
repository is multi user automatically as database is multi user. It
means files can be accessed from all over the world and groupware
collaboration becomes trivial as PostgreSQL is networked database.

Now for the user accessing the specific database then the hyperlink
can be just (hyperscope 432) and user who is remote could say by
activating the hyperlink that remote user likes to have the
file.

Program must know if user is local or remote. If user is remote the
file can be sent by email, it could be automatically encrypted and
sent, encrypted and uploaded to web server, or uploaded to web server
with password encrypted access or without. Any information can be
protected and not all information need to be shown on public
webservers.

Maybe it becomes better to use the URI like:
hyperscope://user@example.com:432

whereby 432 would not indicate the database port but rather the ID of
the hyperdocument to be activated or accessed. user@example.com would
have relation to the actual username, password, hostname, database
name and port on the user's own system and program installed as such
without local database. Maybe port could be optional as multiple ports
could be on the same hostname.

That would create unique access to specific domain and specific user
on remote hyperscope server. It becomes possible to securely share and
access files or do any action on such files by using the groupware
features.

The big difference with the WWW is that system is structured and
offers liberty on how to access files and what to do with the system.

User may remotely invoke emails to be sent to groups of people. This
is not what WWW offers by default.

User could remotely edit Org file only by using the database. Org file
need not be located on the file system. And yet such Org file can be
automatically saved on the file system or sent to other people.

Locating 4-5 or more people becomes possible, marking of them and
quick export to Org file becomes possible. Then user may invoke
further actions such as visiting people, negotiating, calling people,
sending them information by post, sending SMS to people.

When backed up by well networked database it becomes multi user
collaborative meta level Org system. 

> - https://blog.jethro.dev/posts/org_mode_workflow_preview/

I have captured that one for later research. Small details and notes
do matter when creating some new useful features.

Jean

Re: One vs many directories

[Christopher Dimech]

> Sent: Saturday, November 28, 2020 at 5:16 PM
> From: "Jean Louis" <bugs@gnu.support>
> To: "Ihor Radchenko" <yantar92@gmail.com>
> Cc: "Dr. Arne Babenhauserheide" <arne_bab@web.de>, "Texas Cyberthal" <texas.cyberthal@gmail.com>, "emacs-orgmode@gnu.org" <emacs-orgmode@gnu.org>
> Subject: Re: One vs many directories
>
> * Ihor Radchenko <yantar92@gmail.com> [2020-11-24 10:57]:
> > > I find it entertaining for now. Now, what is exomind?
> >
> > Unless I misunderstood, Jean referred to "external brain" concept:
> > - https://beepb00p.xyz/exobrain/
>
> The more you send me reference more I discover other set of people
> doing same what I am doing. Since I have implemented central meta
> level organization it is moving rapidly, everthing gets sorted. It
> develops by itself and is rapidly accessible.

Believing that only you think a certain way is a big mistake.

> That website I have to mirror locally to pick ideas and learn from
> others. Mirroring I do with:
>
> $ wget -Emk http://example.com
>
> As that command replaces all hyperlinks to local hyperlinks. That
> person advanced in organization of things. I stick to few principles
> and just design it by principles.
>
> Design works rapidly. Few Emacs Lisp functions and access to reports
> listed in Emacs Buffers and integration with other tools.
>
> With one function and one PostgreSQL table defined in 3 minutes I get
> rudimentary backup and version system for any column values that I am
> editing in the database. If I edit note, the note is versioned
> (previous version stored) before I start editing it. Principles I am
> following are basics what programmers like, to minimize or eliminate
> repetitions and efforts to achieve the goal.
>
> Person above have extracted or exported its own database of hyperlinks
> to hyperdocuments. My side I have made for now Org export of any
> subtree or the whole dynamic knowledge repository. There are many
> things to go. In Emacs development version all kinds of hyperlinks can
> get their handlers like gopher:// gemini:// message: tel: sms: and
> htat will be very helpful.
>
> No, I do not use "exobrain" as a term. I rather lean on Engelbart's
> terminology and follow his principles as we are very late to implement
> what was envisioned back in 1968 and before. It is 52 years already.
>
> And many more years since Memex has been invented:
>
> Memex
> https://en.wikipedia.org/wiki/Memex
>
> As author said: "The memex device as described by Bush "would use
> microfilm storage, dry photography, and analog computing to give
> postwar scholars access to a huge, indexed repository of knowledge any
> section of which could be called up with a few keystrokes."
>
> And that is exactly what I am creating here to have anything called up
> with few keystrokes and to be able to share files with individuals or
> groups of people without more thinking but just designated what to be
> done.
>
> Have group of 5 people to share notes with? Just find the designated
> group and click share. Computer would handle the rest, maybe send
> files by emails individually, maybe inform people by SMS, maybe upload
> files and share password protected hyperlinks with those people.
>
> Integration is another keyword I like to follow. Android principle of
> sharing is pretty much based on integration. We have all the small
> functions around us only not well integrated with their relations that
> concern human problems.
>
> We have files on file system which we cannot easily share with groups
> or people we want. Address books are all sparse, one is in this email
> client, one is separate, one is on the mobile device, another email
> client does not synchronize, and so on. I have forgotten this long
> ago and use central address book from where everything derive:
>
> - no Google, clouds, etc. that is very insecure. Do not give contacts
>   to Google, there are hundreds of thousands of staff members there
>   and no guarantee whatsoever that they will not read it.
>
> - keeping contacts on my computers. I have already spent money for
>   hard disk, there is enough space
>
> - exporting contacts from central database and importing to email
>   clients, mobile devices, this way everything is synchronized.
>
> How quickly can GNU/Linux user share a file with somebody?
>
> - locate the file by using hierarchical browsing. If file system is a
>   mess, this alone may take some time
>
> - open up email reader
>
> - find that email address. If it is in the email reader already it is
>   good. But it could be in the phone. It could be on paper, or on
>   business card. Where is it? Maybe calling person? But where is the
>   phone number? On first phone, second phone... if all is synchronized
>   maybe is easier to find.
>
> - attach the file
>
> - send the file.
>
> But then sending SMS or calling in the same time does not
> work. The above process is not well integrated.
>
> It could work like this:
>
> - user just thinks of what has to be shared with other person, types
>   the terms related to the thought
>
> - locates the file and press share
>
> - locates the user and press enter. FINISHED
>
> That would be better integration. Even better it would be if user can
> choose the automated workflow option:
>
> 1. send the file, automatically record that file has been sent to
>    specific user. Tell user automatically how many files are attached
>    and attach annotation belonging to the file as body of the email or
>    any instructions.
>
> 2. in the same time inform the user by SMS that file has been sent and
>    record that SMS have been sent. Software like kdeconnect, gnokii
>    can be used for it.
>
> 3. within 1 hour, or other period of time, computer asks to initiate
>    the call to the user to follow up about the file sent and maybe
>    nudges few times and records the action. Software like termux tools
>    can be used for it.
>
> My first big surprise with Org was that there was no possibility to
> assign the task to other person and send that task! I actually could
> not believe that it was meant for single person or personal tasks and
> notes. Then I made the function to share the task quickly to any
> person assigned to the task. If person is assigned, task is sent to
> the person. If no person is assigned then I choose to which person to
> send it. This includes also groups of people.
>
> > - https://zettelkasten.de/posts/extend-your-mind-and-memory-with-a-zettelkasten/
>
> That is similar idea of organizing. There is claim that one shall
> forget about categories and rather use tags. I think that using any
> types of attributes is better and using more attributes helps in
> quicker location.
>
> > - https://github.com/novoid/Memacs
>
> I have installed it and not yet used it. I would not like having too
> many tools on file system to manage information. There are too many
> memacs tools made for console. In general I am tracking all SMS sent
> from phones to other people, they are automatically inserted into
> corresponding people's objects. Then I know which people received
> what. Phone calls can be tracked too. Phone calls can be
> recorded. Sales and marketing departments need that. I am using now
> only principles from Memacs and implement some of them in Emacs
> Lisp. As I like integration I do not like external tools, but the
> dynanic knowledge repository must be usable externally without
> Emacs.
>
> That becomes very easy by expanding the whole tree of notes into the
> file system from time to time and generating meanings for symlinks
> that point to fixed locations on file system.
>
> The centralized subtrees or nodes of my dynamic knowledge repository
> can be moved easily from one parent node to other parent node. This is
> because human must sort things properly. But if such are pointing to
> files on file system those files never change. Meaning and relations
> can change but file location should not change. Directories are more
> static then files. They would never change. Files if not indexed but
> located in the archive could maybe change or get updated.
>
> Git repository could get updated but its directory need never change
> in the future.
>
> Let us say there are many PDFs to be indexed and accessed through
> semantics. The PDF file name could change but access to PDF file need
> never change. Renaming PDF file need not change access to PDF file in
> other word there is no need to rename it twice, it can remain in the
> database and get accessed automatically. But that requires directory
> to be static, or it requires md5sum of the file to be
> static. Something must be static that file can be found by the
> system. Best is when file is under specific unique ID that never
> changes. Then everything becomes unique and clear. And symlink can be
> automatically generated:
>
> ~/hyperscope/1/2/3/432.pdf would be file
>
> ~/hyperscope/1/2/3/Knowledge.pdf would be symlink to 432.pdf
> automatically generated and from time to time updated if there were
> many changes in the database.
>
> The Org hyperlink to the file could point to:
> ~/hyperscope/1/2/3/432.pdf because file location is this way static
> and will never change.
>
> But the Org hyperlink could as well point to meta level hyperlink
> (hyperscope 432) as that would open the file no matter where is the
> location.
>
> And if file is on remote server, something like
>
> [[PDF File][(hyperscope 432 2)]]
>
> would then work quite well. As this type of dynamic knowledge
> repository is multi user automatically as database is multi user. It
> means files can be accessed from all over the world and groupware
> collaboration becomes trivial as PostgreSQL is networked database.
>
> Now for the user accessing the specific database then the hyperlink
> can be just (hyperscope 432) and user who is remote could say by
> activating the hyperlink that remote user likes to have the
> file.
>
> Program must know if user is local or remote. If user is remote the
> file can be sent by email, it could be automatically encrypted and
> sent, encrypted and uploaded to web server, or uploaded to web server
> with password encrypted access or without. Any information can be
> protected and not all information need to be shown on public
> webservers.
>
> Maybe it becomes better to use the URI like:
> hyperscope://user@example.com:432
>
> whereby 432 would not indicate the database port but rather the ID of
> the hyperdocument to be activated or accessed. user@example.com would
> have relation to the actual username, password, hostname, database
> name and port on the user's own system and program installed as such
> without local database. Maybe port could be optional as multiple ports
> could be on the same hostname.
>
> That would create unique access to specific domain and specific user
> on remote hyperscope server. It becomes possible to securely share and
> access files or do any action on such files by using the groupware
> features.
>
> The big difference with the WWW is that system is structured and
> offers liberty on how to access files and what to do with the system.
>
> User may remotely invoke emails to be sent to groups of people. This
> is not what WWW offers by default.
>
> User could remotely edit Org file only by using the database. Org file
> need not be located on the file system. And yet such Org file can be
> automatically saved on the file system or sent to other people.
>
> Locating 4-5 or more people becomes possible, marking of them and
> quick export to Org file becomes possible. Then user may invoke
> further actions such as visiting people, negotiating, calling people,
> sending them information by post, sending SMS to people.
>
> When backed up by well networked database it becomes multi user
> collaborative meta level Org system.
>
> > - https://blog.jethro.dev/posts/org_mode_workflow_preview/
>
> I have captured that one for later research. Small details and notes
> do matter when creating some new useful features.
>
> Jean
>
>

Re: One vs many directories

[Texas Cyberthal]

Hi Jean,

> Now, what is exomind?

https://cyberthal-docs.nfshost.com/cyborganize/exomind/

What you described is not how you think, it is how you wish your CRM
info retrieval system to perform conveniently.  Almost nobody has a
formal thought algorithm, because brains have ADD compared to
computers.

Textmind is a tool for general human text thoughts.  It's unspecialized.

If you have a huge number of incoming text of a special type, such as
customer leads, who are just cogs that you don't genuinely ponder,
then Textmind is the wrong tool for that job.  You need SME CRM
software.

Just common sense.  Combine harvester works "better" than a push
lawnmower, but everyone still needs a push lawnmower.

Ok, Textmind feels more like a riding lawnmower to me, because so
comfy, but also better detail work, like a wheedwhacker... and the
analogy falls apart.

> Contract signed between your company 123 and person ABC?

I might like a separate Textmind for a company.  If separate, then
file under ~/1-Mansort/1-Textmind/2-Linked/7-Name/2-Flat/Doe-John~.
Otherwise, sounds like something a relational database should handle.

> Image on which there is only your family member, not you, which has its date?

Same path, but ~/Surname-/Given-name~, and binaries are stored in Binmind.

> Image of you, with the date?

Same path, but ~/2-Me~.

- Image without date in the file name and not embedded?

Depends on image content.

- Software git tree related to mailing things?

~/1-Mansort/1-Textmind/2-Linked/2-Codex/~

Large physical keyboards are the best human to computer input device
for work and nothing on the horizon challenges that.  Keyboards are
spreading to more enterprise contexts, such as keyboards in cop cars,
as software eats everything.

> The other complete thought algorithm is Pubmind, for longform content.
> But it doesn't work without Textmind.

https://cyberthal-docs.nfshost.com/cyborganize/pubmind/

I suppose it takes Textmind+Pubmind to make a complete thought
algorithm.  Pubmind terminates in publications rather than action.
Pubmind is only necessary once Textmind starts growing clogged with
publications that no longer belong inside one's personal exomind.

Re: One vs many directories

[Jean Louis]

* Texas Cyberthal <texas.cyberthal@gmail.com> [2020-11-25 09:58]:
> Hi Jean,
> 
> > Now, what is exomind?
> 
> https://cyberthal-docs.nfshost.com/cyborganize/exomind/

How I like those thoughts. 

> Mind vs brain 
> Your brain is the squishy jelly between your ears. 
> Your mind is what disappears when that jelly gets shaken too hard by
> a punch.

There are those who die and come back and view things from above and
can think and use their mind even though brain was turned off
temporarily.

> Exomind — an individual's PIMS and the personal info it contains. One
> can wax philosophical about how far the exomind extends into situated
> cognition. Perhaps the Internet is a shared global exomind. Anyway,
> Cyborganize is a personal exomind.

I just get idea how exoskelet does that.

You also spoke of device, do you really mean physical device?

> What you described is not how you think, it is how you wish your CRM
> info retrieval system to perform conveniently.  Almost nobody has a
> formal thought algorithm, because brains have ADD compared to
> computers.

I cannot get it as I do not know what is ADD.

> If you have a huge number of incoming text of a special type, such
> as customer leads, who are just cogs that you don't genuinely
> ponder, then Textmind is the wrong tool for that job.  You need SME
> CRM software.

But I have M-x already that works.

Thank you for examples of file sorting with 10 Bins.

> https://cyberthal-docs.nfshost.com/cyborganize/pubmind/

Thank you, I was reading.

Re: One vs many directories

[Texas Cyberthal]

Hi Jean,

> There are those who die and come back and view things from above and can think and use their mind even though brain was turned off temporarily.

I didn't say that the mind always turns off when the brain is damaged.

> You also spoke of device, do you really mean physical device?

Brain-Computer Interface is a term that usually means an
electromagnetic connection between nerves and electronics.  However,
really a keyboard is a superior version of that for non-motor tasks,
despite being insulated on both ends with skin and plastic.  Human
fingers are amazingly dextrous, so their bandwidth is high.
Eventually mammalian brain plasticity will permit implants to surpass
keyboards, but that will take a while and might require implantation
as a child to achieve maximum results.  The generational technology
gap moves ever inward, towards our very genes.  Obsolete thyself to
perpetuate thy meme-ery.

> I cannot get it as I do not know what is ADD.

Attention Deficit Disorder

> But I have M-x already that works.

Because you built your own SME CRM.  My point is that an SME CRM
should probably be a separate entity from a Textmind.  Their purposes
conflict.

Re: One vs many directories

[Jean Louis]

* Texas Cyberthal <texas.cyberthal@gmail.com> [2020-11-25 13:40]:
> > You also spoke of device, do you really mean physical device?
> 
> Brain-Computer Interface is a term that usually means an
> electromagnetic connection between nerves and electronics.  However,
> really a keyboard is a superior version of that for non-motor tasks,
> despite being insulated on both ends with skin and plastic.

hahahha you are right

> Because you built your own SME CRM.  My point is that an SME CRM
> should probably be a separate entity from a Textmind.  Their
> purposes conflict.

Really, but for me those similarities extend each other. Not that I am
sorting files by Textmind but I do have similar method in sorting
files. You sort files by tags in a hierachy, that is the concept of
Textmind. How tags are called or how they relate to each other is then
rather implementation.

Anyway I wish to know more of that, subscribe me to your mailing
list. In last days I got more ideas than I was thinking I will, all
from this Org mailing list.

And now slowly I start implementing.

I like Org but I also like more structured data.

Re: One vs many directories

[Texas Cyberthal]

Hi Jean,

Yes, Textmind is a rock tumbler for natural language thoughts.  An SME
CRM treats people like widgets.  The former does many small thoughtful
touches, the latter does few robotic touches.  Excessive widget volume
chokes Textmind.

Sure, I will subscribe you when I have a mailing list.  For now,
readers can only follow the Github repos.

Re: One vs many directories

[Jean Louis]

* Texas Cyberthal <texas.cyberthal@gmail.com> [2020-11-26 19:05]:
> Hi Jean,
> 
> Yes, Textmind is a rock tumbler for natural language thoughts.  An SME
> CRM treats people like widgets.  The former does many small thoughtful
> touches, the latter does few robotic touches.  Excessive widget volume
> chokes Textmind.
> 
> Sure, I will subscribe you when I have a mailing list.  For now,
> readers can only follow the Github repos.

Alright.

I can see there are few people who have got a cognition in organizing
things not being only Org notes but also files, and I feel you are one
of them. I would like to ask few questions:

- does using the 10 Bins and Textmind system gives you personal
  satisfaction of being well organized?

- did you develop having functions similar to store link that quickly
  obtain the hyperlink in memory to be easier inserted in Org files?
  That is similar to org-capture. I think every system of organization
  and storing objects into X should have automated quick hyperlink
  generation.

- how does 10 Bins and Textmind enhance what you do with Org files?

Jean

Re: One vs many directories

[Texas Cyberthal]

Hi Jean,

> does using the 10 Bins and Textmind system gives you personal satisfaction of being well organized?

For what it does, yes, amazingly so.  I still need Dbmind, which I
haven't developed yet.

> did you develop having functions similar to store link that quickly obtain the hyperlink in memory to be easier inserted in Org files? That is similar to org-capture. I think every system of organization and storing objects into X should have automated quick hyperlink generation.

I find Emacs Org's native facilities adequate.  However, I did a bit
of streamlining:

> C-c l runs the command treefactor-org-store-link-fold-drawer (found in global-map), which is in ‘treefactor.el’.

> how does 10 Bins and Textmind enhance what you do with Org files?

It mind-syncs a natural language thought algorithm, which would
otherwise be impossible.

https://www.geeksforgeeks.org/introduction-to-algorithms/

It is clear and unambiguous, has well-defined inputs and outputs, and
is finite and feasible.

Unlike Getting Things Done by David Allen, it captures the whole
thought-stream, or at least everything worth typing.

Man is used to thinking alone, with no internal error-checker.  Sloppy
conclusions abound since biological memory is ephemeral.  Textmind
creates a team of past and future selves to collaborate
asynchronously.  It's quite steadying.

Re: One vs many directories

[Jean Louis]

* Texas Cyberthal <texas.cyberthal@gmail.com> [2020-11-27 12:01]:
> Hi Jean,
> 
> > does using the 10 Bins and Textmind system gives you personal
> > satisfaction of being well organized?
> 
> For what it does, yes, amazingly so.

Thank you. I was expecting something like that as we are in similar
position of having somewhat personally better or could we say
idiosyncratically better organizing system for ourselves.

> I still need Dbmind, which I haven't developed yet.

What should it be or do?

> > did you develop having functions similar to store link that
> > quickly obtain the hyperlink in memory to be easier inserted in
> > Org files? That is similar to org-capture. I think every system of
> > organization and storing objects into X should have automated
> > quick hyperlink generation.

> I find Emacs Org's native facilities adequate.  However, I did a bit
> of streamlining:
> 
> > C-c l runs the command treefactor-org-store-link-fold-drawer
> > (found in global-map), which is in ‘treefactor.el’.

As you have specific thought order in directory names then maybe such
could be parsed, maybe slashes / removed to show a full path to the
file. This becomes long but could be useful in some lists.

> > how does 10 Bins and Textmind enhance what you do with Org files?
> 
> It mind-syncs a natural language thought algorithm, which would
> otherwise be impossible.
> 
> https://www.geeksforgeeks.org/introduction-to-algorithms/
> 
> It is clear and unambiguous, has well-defined inputs and outputs, and
> is finite and feasible.

Alright and I find that it is the case on my side, and previous work
of Engelbart, then also within some other information management
systems, like Semantic Synchrony.

Would you consider that the system I am using would be or could be
said to have natural thought algorithm by considering following:

That there are many nodes, they have their names, some properties, and
tags and other meta data and searchable lines could look like this as
one variety of accessing methods:

People / Jean / Computer / Free Software / GNU Emacs / Real-time incremental narrowing completion / Helm

now imagine 20,000 or 100,000 such lines but not visible to user, just
few would be visible as incremental narrowing search such as helm or
ivy or filtering functions could quickly locate particular lines.

If I think of "GNU" I would get maybe subset of lines related to that
thought, if I think of Emacs, I would get references related to GNU
and Emacs. If I think of "completion" I would get references to ivy,
selectrym, helm, and so on and by watching the dynamical filter in
front of me I get new references displayed in real time which I can
then add to the query until I find the matching few or matching tree
or matching node.

Sounds to me it is also one type of natural thought algorithm.

> Unlike Getting Things Done by David Allen, it captures the whole
> thought-stream, or at least everything worth typing.

I have no idea and by reading basic descriptions I do not find myself
there. It seems that systems are pretty much idiosyncratic unless
training and well written instructions help in making it applicable
for a group.

There is one "algorithm" that I am using so that every task CAN BE
DONE, and that is that tactical plans are divided into projects only
when one step of the plan is too complex to be executed without
dividing it or chunking it down. Projects fullfil one step of a larger
plan and consists of steps. When one step cannot be fullfiled by its
own, it probably means it was not written well chunked, so that step
is chunked into tasks or orders. Principle of the algorithm would be
to never write tasks that are too complex to be executed and finalized
and to have each task in itself doabl so that each step becomes simple
step of one bigger complex action.

We do projects in real life such as purchasing, construction of
machinery, negotiations, and so on, and projects are written on the
paper and executed and again reused and executed on other places. By
following the above principle our staff members in various countries
could get things done just by reading instructions as each step has
been simplified down to the atomic doable task.

Re: One vs many directories

[Texas Cyberthal]

Hi Jean,

> What should it be or do?

Dbmind does things that Postgres handles better than Org.

> As you have specific thought order in directory names then maybe such could be parsed, maybe slashes / removed to show a full path to the file. This becomes long but could be useful in some lists.

I don't intend to do so.  Textmind maximizes path dynamism via
Dired+Treefactor.  Links shouldn't break that.

> Alright and I find that it is the case on my side, and previous work of Engelbart, then also within some other information management systems, like Semantic Synchrony.

Some of that might qualify as an algorithm, but not a natural thought
algorithm.  A natural thought algorithm must manage substantially all
natural thoughts while satisfying the definition of an algorithm.

The things you mentioned are not even as sophisticated and complete as
GTD.  And GTD is merely a personal paper-management algorithm, not a
natural thought algorithm.

(Textmind manages text only, whereas some people think visually.  It
would be easy to adapt Textmind principles to Binmind, if needed.
Therefore even for visual thinkers, Cyborganize is a natural thought
algorithm.)

I doubt there are multiple ways to design a natural thought algorithm.
For example, all natural thoughts occur in a chronological sequence.
This necessitates a ramblog to accurately reflect them.

This is the GTD inbox algorithm:

https://michaelwhatcott.com/gtd-workflow-processing-algorithm/

GTD is usually called a method.  I've started calling Textmind an
algorithm to emphasize the finiteness aspect of its design, a key
feature.  I think I could construct a pseudocode Textmind algorithm.
It would of course rely on human judgment for some decisions, and
judgment is fuzzy.  But the algorithm itself would be unambiguous.

Re: One vs many directories

[Jean Louis]

* Texas Cyberthal <texas.cyberthal@gmail.com> [2020-11-28 11:20]:
> Hi Jean,
> 
> > What should it be or do?
> 
> Dbmind does things that Postgres handles better than Org.
> 
> > As you have specific thought order in directory names then maybe
> > such could be parsed, maybe slashes / removed to show a full path
> > to the file. This becomes long but could be useful in some lists.

> I don't intend to do so.  Textmind maximizes path dynamism via
> Dired+Treefactor.  Links shouldn't break that.

I enjoy these thoughts.

Maybe you refer to browsing the tree.

I am switching completely from browsing the tree into index type of
access. Index helps to find the thought which in turn finds candidates
and file is in front of me. How I think now is little different than
past and different than future. If I think of term "Insurance" related
to Germany those are two words to search for. Maybe in future I think
of "health" then I should be able to find "insurance". There is no
fixed pattern on how to think. Repetition in locating the node creates
habits that next search is even quicker found.

Those most searched nodes could be ranked automatically for even
quicker access. And all searches for nodes could be recorded for
future as inexpensive and automated log that helps person find what
happened in the history and which nodes have been located at certain
months, dates, years.

org-store-link would get its hook or additional function that each
time when Org mode related heading is is stored that it ranks the
file the heading in a separate list which can be stored on file
system.

After a while user will get a subset of highly ranked headings in
their corresponding Org files. That subset then can be used as quick
bookmarks or get bound to keys.

> > Alright and I find that it is the case on my side, and previous work of Engelbart, then also within some other information management systems, like Semantic Synchrony.
> 
> Some of that might qualify as an algorithm, but not a natural thought
> algorithm.  A natural thought algorithm must manage substantially all
> natural thoughts while satisfying the definition of an algorithm.

I wish I could understand definition of "natural though algorithm" in
the context how you refer it to.

From other people's experiences I can see they are thinking
different. It is questionable if there is one algorithm corresponding
to many people's natural thinking.

The algorithm in locating specific file on my sides is programming
algorithm based on what I find most quick for me personally. There are
several such algorithms programmed that help me locate things.

I have 19489 nodes, everything is together. There is no visible delay
for completion to show me list of nodes. The list of everything is
offered for completion and I can type what I think that I need.

If I search for PDF related to "mercury" I will locate it as quick as
1-2 seconds and can open it already. Or I could search by word, tag,
attribute and get a list of related hyperlinks and
hyperdocuments. From there on I can locate the right one.

> The things you mentioned are not even as sophisticated and complete as
> GTD.  And GTD is merely a personal paper-management algorithm, not a
> natural thought algorithm.

I did look on the hyperlink you gave me. It looks like algorithm
deciding what to do with tasks for me. But that is not how we manage
tasks in the group.

We have processes that are defined from A to Z and tasks are managed
with to achieve the overall purpose. A purpose could be project
accomplishment such as "bridge built over the water". Then tasks are
steps bringing the project closer to accomplishment, such as purchase
timber, saw, meter tape, screws. Project planner is the key here as
one should not put nothing more and nothing less, and none task shall
be defined that is not doable. No algorithm should influence tasks to
be pending, or archive them or decide to do task if it is very short
as relations to task and from the task to/from other objects are not
so simple. There can be 10 tasks each 2 minutes long that cannot be
conducted right now as the phone call and subsequent travel may have
so much more important.

I was referring to this:

> After reading appendix IV to Making it all work (another book about
> GTD by David Allen) it occured to me that the Process phase of
> "Mastering Workflow" would be well-rendered using psuedocode. This
> is probably due to the fact that there are several decisions to make
> along the way (logic) and this happens for each item in your inbox
> (loop). So, without further introduction here's the code:

> while length(inbox) > 0:
> 	inbox_item = inbox.pop()
> 	thing = analyze(inbox_item)

> 	if not actionable(thing):
> 		toss(thing) or tickle(thing) or file(thing)

> 	else if less_than_two_minutes(thing):
> 		do(thing)

> 	else if is_single_task(thing):
> 		wait_for(delegate(thing)) or assign_context(thing)

> 	else:
> 		create_new_project(thing)

If the bridge gets built, all the undone tasks become redundant. The
algorithm shown tries to manage something but would be detrimental to
the actual task and project accomplishment we are doing here. Overall
this discussion does help me to see your insights and thoughts, other
people thoughts and to build upon it and create new ideas and
implementations. 

> I doubt there are multiple ways to design a natural thought
> algorithm.  For example, all natural thoughts occur in a
> chronological sequence.  This necessitates a ramblog to accurately
> reflect them.

Memories are saved chronologically. By thinking what I was about doing
in the years about 2004, I can jump to images sorted chronologically
and access maybe image in 2005 as first thought was maybe not so
precise. But it is about there.

But I cannot access image related to business ABC that is located in
2004 quickly unless such image is indexed somewhere by its meaning,
maybe "boat purchased" would be meaning related to such picture. Then
if I wish to see boats I have purchased I would type most probably
"boat" and from there find various other attributes such as dates,
types and similar.

> This is the GTD inbox algorithm:
> 
> https://michaelwhatcott.com/gtd-workflow-processing-algorithm/

I cannot relate it as useful for work we are doing. And we did
accomplish many projects successfully. Principle is like this:

* Plan

- purposes
- goals

** Plan steps

1. do
2. do

If plan step cannot be easily conducted, it has to be branched into
Project.

* Plan

- purposes
- goals

** Plan steps

*** Project 1. do

- task 1
- task 2

*** Project 2. do

- task 1
- task 2

If any purpose get accomplished, the undone tasks become redundant and
can be forgotten, but could be reused in future.

> GTD is usually called a method.  I've started calling Textmind an
> algorithm to emphasize the finiteness aspect of its design, a key
> feature.  I think I could construct a pseudocode Textmind algorithm.
> It would of course rely on human judgment for some decisions, and
> judgment is fuzzy.  But the algorithm itself would be unambiguous.

I have fully understand concepts of 10 Bin. But I cannot adapt to the
thinking pattern that I did not yet understood. I did read but did not
understand it.

Since we talked about it I have developed more so that I can soon
forget about the file system and only use conceptual access and
filing. I am in transition of all files sorted hierarchically to be
sorted semantically while program is to make sure of hierarchies.

Let us say video is there, I can file it by choosing the genre which
is node name and place some interesting tags or attributes, maybe
regisseur, movie director if necessary, there are movie researchers
who do exactly that. Once necessary attributes are there it becomes
pleasure later to access it in future and play it or send to
somebody. But there is no need any more to hierarchically browse the
tree of directories until movie is found. Any related concept will
lead by associations to the movie. If one does not remember the movie
name, maybe remembers movie director or actor or maybe genre or even
approximate movie year or other attribute.

If all associations such as node name, attributes, tags, etc are
indexed and quickly accessible (similar to database view but rather
table with the index to get blazing fast query) -- then locating any
item among huge list of items becomes rapid.

Think of a concept what you want to get? To play movie related to
"eagle"? Type word and get "Where the Eagles Dare", but maybe one has
to type also "movie" to distinguish it from subject of birds.

Think of a concept what you want to file? Designate the type, is it
movie, PDF, meeting, TODO task, note, survey, designate place, date,
group of people, person or other attributes or in other words
integrate the relations of a hyperdocument. From there on, forget
about it.

Mnemonics techniques use exactly the same conceptual and associative
remembering.

Number 2 is often associated to a swan. It should be swan doing some
action, for example, swan that is eating something where something is
the object to be remembered.

Let us say that object number 2 is bucket of metal screws. All one has
to do is to connect relations with the fixed relation of swan on the
number two that is eating something.

Then it becomes a swan drowining in the lake as swan has eaten a full
bucket of screws which shape can be seen on its neck. Just imagine the
picture as picture itself is relational object as associations are
relations. Forget about the picture.

When it comes to tell which object was at the place number 2, one
associates first 2 to swan, and immediately remembers the drowning,
the lake and that screws in the bucket were heavy and that is the
object that had to be located.

Re: One vs many directories

[Texas Cyberthal]

Hi Jean,

> After a while user will get a subset of highly ranked headings in their corresponding Org files. That subset then can be used as quick bookmarks or get bound to keys.

This is a higher tier of PIM than Textmind.  Textmind is for
processing thoughts.  For example, I use it to convert these email
exchanges into useful notes and documentation.

Crystallized knowledge that needs to be optimized for rapid or lateral
access belongs in a PKM or CRM app.  There are many options, and which
is best likely depends on the specific use case.  (Assuming it doesn't
belong in Postgres or Pubmind.)

> From other people's experiences I can see they are thinking different. It is questionable if there is one algorithm corresponding to many people's natural thinking.

Almost nobody has a holistic natural thought algorithm.  An algorithm
has a strict definition, whereas biological thought is evolved and
messy.

> But I cannot access image related to business ABC that is located in 2004 quickly unless such image is indexed somewhere by its meaning, maybe "boat purchased" would be meaning related to such picture. Then if I wish to see boats I have purchased I would type most probably "boat" and from there find various other attributes such as dates, types and similar.

Images should have a creation date in metadata and also some tags, I
agree.  Whether to organize them in a Binmind using 10 Bins is a
matter of taste.  I prefer to keep them in 10 Bins until their tag
nomenclature is mature.  Then, when worthwhile, I would move them to a
dedicated image management app, and add a good set of tags and
metadata to ensure they won't be lost later.

If the app allows it, I might still organize the pics into a tree,
just to make it easier to navigate the whole collection without
repetition, and understand its primary meaning quickly.  This extra
work is profitable only for high-value images.  The existence of the
tree permits continued nomenclature evolution without risking some
images getting lost due to tag drift and rot.

Treemind is good at nomenclature evolution via Rapid Inductive
Iterative Tree Refiling.

Re: One vs many directories

[Jean Louis]

* Texas Cyberthal <texas.cyberthal@gmail.com> [2020-11-29 09:20]:
> Images should have a creation date in metadata and also some tags, I
> agree.  Whether to organize them in a Binmind using 10 Bins is a
> matter of taste.  I prefer to keep them in 10 Bins until their tag
> nomenclature is mature.  Then, when worthwhile, I would move them to a
> dedicated image management app, and add a good set of tags and
> metadata to ensure they won't be lost later.
> 
> If the app allows it, I might still organize the pics into a tree,
> just to make it easier to navigate the whole collection without
> repetition, and understand its primary meaning quickly.  This extra
> work is profitable only for high-value images.  The existence of the
> tree permits continued nomenclature evolution without risking some
> images getting lost due to tag drift and rot.

That gives me idea for Emacs similar to org-noter that one could have
bunch of pictures being displayed then just choosed few tags by using
mouse and window get replaced with new picture. Process could be used
for faster tagging.

> Treemind is good at nomenclature evolution via Rapid Inductive
> Iterative Tree Refiling.

For that I need video to understand.

Re: One vs many directories

[Texas Cyberthal]

Hi Jean,

> For that I need video to understand.

Agreed.  I thought the Treefactor gif videos would be enough, but it's
clear that people's imagination cannot extrapolate the utility of
RIITR.

I developed this skill long ago on the Windows app Brainstorm, and
have forgotten how rare and unintuitive it is.

This is the key missing concept preventing people from understanding
and adopting Textmind, because Textmind is designed primarily to
exploit this concept.

So I plan to play AI Dungeon with Treefactor, using RIITR to make
sense and fun of nonsense, and record it on YouTube with live audio
commentary.  Monkey see, monkey do.  Then people will be able to do
RIITR.

Re: One vs many directories

[Ihor Radchenko]

Texas Cyberthal <texas.cyberthal@gmail.com> writes:
> Agreed.  I thought the Treefactor gif videos would be enough, but it's
> clear that people's imagination cannot extrapolate the utility of
> RIITR.

That would be appreciated. I tried to read Treefactor docs at least 3
times and failed to understand its utility.

Best,
Ihor

Re: One vs many directories

[Texas Cyberthal]

Ihor> That would be appreciated. I tried to read Treefactor docs at
least 3 times and failed to understand its utility.

Then I will move AI Dungeon Treefactor demo priority above all but
critical Cyborganize documentation, such as broken links.

Apparently I've wasted a lot of time documenting Textmind and 10 Bins
before Treefactor was adequately explained.

To avoid such mistakes in the future, I should focus on instilling the
minimum viable behavioral change.

Untangling a cognitive knot with an impromptu Treefactor or
Brainstormsw session is a good candidate:

https://www.brainstormsw.com

And recreational use, such as for gaming, is a sticky alternative to
undesirable premature production use.

Re: One vs many directories

[Jean Louis]

* Texas Cyberthal <texas.cyberthal@gmail.com> [2020-11-30 10:36]:
> Hi Jean,
> 
> > For that I need video to understand.
> 
> Agreed.  I thought the Treefactor gif videos would be enough, but it's
> clear that people's imagination cannot extrapolate the utility of
> RIITR.
> 
> I developed this skill long ago on the Windows app Brainstorm, and
> have forgotten how rare and unintuitive it is.
> 
> This is the key missing concept preventing people from understanding
> and adopting Textmind, because Textmind is designed primarily to
> exploit this concept.
> 
> So I plan to play AI Dungeon with Treefactor, using RIITR to make
> sense and fun of nonsense, and record it on YouTube with live audio
> commentary.  Monkey see, monkey do.  Then people will be able to do
> RIITR.

You could record on some of free hostings that respect users' freedom
that refrain of coercing non-free javascript such as:

Open.tube upload
https://open.tube/videos/upload

Re: One vs many directories

[Ihor Radchenko]

Jean Louis <bugs@gnu.support> writes:
> You could record on some of free hostings that respect users' freedom
> that refrain of coercing non-free javascript such as:
>
> Open.tube upload
> https://open.tube/videos/upload

Thanks for this reference.

You also mentioned hypothes.is earlier. Do you know if there is any
similar software allowing to store data locally without a need to create
account and trust the service for storing all the personal notes?

Best,
Ihor

Re: One vs many directories

[Ihor Radchenko]

Jean Louis <bugs@gnu.support> writes:
> You could record on some of free hostings that respect users' freedom
> that refrain of coercing non-free javascript such as:
>
> Open.tube upload
> https://open.tube/videos/upload

Thanks for this reference.

You also mentioned hypothes.is earlier. Do you know if there is any
similar software allowing to store data locally without a need to create
account and trust the service for storing all the personal notes?

Best,
Ihor

Re: One vs many directories

[Jean Louis]

* Ihor Radchenko <yantar92@gmail.com> [2020-11-30 15:25]:
> Jean Louis <bugs@gnu.support> writes:
> > You could record on some of free hostings that respect users' freedom
> > that refrain of coercing non-free javascript such as:
> >
> > Open.tube upload
> > https://open.tube/videos/upload
> 
> Thanks for this reference.
> 
> You also mentioned hypothes.is earlier. Do you know if there is any
> similar software allowing to store data locally without a need to create
> account and trust the service for storing all the personal notes?

hypothes.is is free software that may be installed locally.

https://github.com/hypothesis

Re: One vs many directories

[Ihor Radchenko]

Jean Louis <bugs@gnu.support> writes:
> hypothes.is is free software that may be installed locally.
>
> https://github.com/hypothesis

Thanks. You inspired me to try installing it locally again.

The main problem with hypothesis is that it is designed to be deployed
to a server and not to a local machine. Thus, the instructions imply
some knowledge of server configuration and skip several important steps,
which were not obvious for me without spending quite a significant time
googling for various issues during and after the installation.

Anyway, I finally managed to get it working with my local setup. Next
step is Emacs integration...

Best,
Ihor

Re: One vs many directories

[Jean Louis]

* Ihor Radchenko <yantar92@gmail.com> [2020-12-02 05:53]:
> 
> Jean Louis <bugs@gnu.support> writes:
> > hypothes.is is free software that may be installed locally.
> >
> > https://github.com/hypothesis
> 
> Thanks. You inspired me to try installing it locally again.
> 
> The main problem with hypothesis is that it is designed to be deployed
> to a server and not to a local machine. Thus, the instructions imply
> some knowledge of server configuration and skip several important steps,
> which were not obvious for me without spending quite a significant time
> googling for various issues during and after the installation.
> 
> Anyway, I finally managed to get it working with my local setup. Next
> step is Emacs integration...

That is great to hear. I wonder how it can be integrated with Emacs as
it is all Javascript based.

Here is the Org version of: 

Technology Template Project OHS Framework as Org file
http://ix.io/2GdW

I suggest when making tools for Emacs that developer of the tool looks
into the template for open hyperdocument systems that system are
developed that they benefit most people.

- "Open" :: implies vendor-independent access to the hyperdocuments
  within and across work groups, platforms, and applications.

If such system is developed with the database backing and generic
hyperlinks then database is vendor-independent (mostly, but it is
still PostgreSQL, but could be any other because it is SQL), then it
can work for any kind of groups, it can be accessed from various
platforms by using Emacs, but it can also be accessed by browser, but
also through other programming languages would such program be
re-written.

For Org, to be open hyperdocument system it would mean to spread it
onto other editors such as Vim where basic Org mode already exists and
other editors, and applications, such as those on Android like Orgzly
that supports Org. But all that does not make Org yet open especially
in relation to hyperlinks.

Hyperlinks should be dynamic not static and centrally managed not user
managed to be work over various platforms, applications, for various
groups.

Using hyperlinks like:

file:///something/pdf:page 1

Is not portable for many. Such hyperlink should be dynamic, which
means if user accesses the database from remote host such could not
access such hyperlink. It would need to convert to ssh:// or scp:// or
ftp:// or https:// hyperlink or something else.

These types of hyperlinks would be better:

user@hostname.com:PDF:Page:1
or
user@hostname.com:PDF:Query:Title
or
user@hostname.com:PDF:Selection:1,2,3,4

but then again they are not enough generic. I am now making hyperlinks
whihc are very generic, they can just designate the node number and
server from the server list, could be something like:

hyperscope:show:1:2 - to get the human designation of the hyperlink,
                     what it is really.
		     
hyperscope:activate:1:2 - to go to server 1, node 2 and activate
                          hyperlink. If I am on local file, Dired
                          would open, but if user is accessing it
                          remotely the hyperdocument would open in
                          different way. That is what I mean with
                          dynamic hyperlinks.
			  
hyperscope:email:1:2: - it would send it by email to the user. Because
                        system identifies users with their usernames
                        and other attributes

hyperscope:email:1:2:user@example.com - it would send hyperdocument to
                                        other user if there are enough
                                        permissions

In that sense of having generic hyperlinks one should also make
annotations. If I make annotation on Emacs than such should be
translatable to let us say Hypothes.is but if not with that tool,
hyperlink for annotation could open on Emacs with Xpdf or Evince
reader on specific page number while showing annotation in Emacs.

Systems should work equally well in any browser because of their
generic nature. Hypothes.is is now working on browsers, which is for
me very limiting.

They should be liberated from browsers as well. That means that API
should be there that allows access through any interface.

Re: One vs many directories

[Ihor Radchenko]

Jean Louis <bugs@gnu.support> writes:
> That is great to hear. I wonder how it can be integrated with Emacs as
> it is all Javascript based.

The server is mostly python-based (see Language stats in
https://github.com/hypothesis/h). They provide HTTP API:
https://h.readthedocs.io/en/latest/api-reference/. There is even elisp
package utilising that API:
https://github.com/Kungsgeten/hypothesis/blob/master/hypothesis.el

Best,
Ihor

Re: One vs many directories

[Jean Louis]

* Dr. Arne Babenhauserheide <arne_bab@web.de> [2020-11-21 18:04]:
> 
> Jean Louis <bugs@gnu.support> writes:
> 
> > When there are more than 2000 people related notes, tasks,
> > calculations, questions arise if such better be kept in one Org file
> > or multiple Org files in one directory or multiple directories for
> > multiple Org files?!
> 
> This came up multiple times in discussions. I think that it is wrong,
> and the reason comes down to efficiency. If you want to work
> efficiently, then sub-second delays matter, and having 4 files with
> 500 entries means that to search in them you need to open 4 files.

Hallo Dr. Arne,

It maybe wrong and it depends of the approach. My approach is that I
think with people and subjects, not with notes only.

Subject can be special plan like ABC.org and I do not need to search
notes related to that plan outside of ABC, because I do not mix
things. I am searching within one file only.

Things TODO are per subject or per person.

Files pertaining to any person are filed in the person's folder.

Somebody else deals only with personal notes and they maybe put such
in various files and of course they need to search.

I am thinking of "Joe Doe" and here is the flow:

- press s-c (for contacts)

- enter Joe Doe or Joe Doe, Berlin, etc.

- among many Joe Doe, I may narrow down to right one

- click F4 there is Org file for Joe Doe, enter Tasks, Transactions
  and whatever else, send Tasks, Notes to Joe Doe, collaborate or make
  agreements. I never construct or open file for a person, function is
  doing that. It makes the file
  ~/Work/People/By-ID/320431/320431.org

  If I need to search, I search inside of the file. 

- click F5 and find all other files for Joe Doe. For example contracts
  and similar. If I need to search there then I use find and grep and
  similar tools. No need for indexing. Files are mostly sorted by data
  how they come.

There is same flow if I think of a group of people with the difference
that if I need a person I still need to find the person in the list of
people. 

So in general I never need to use some general search through Org
files or any other files as my way of thinking begins with People or
Groups and that narrows what has to be searched.

> If you have 100 files with 20 notes each, you have to open 100
> files.

You maybe mean opening automatically files and searching through
such. I do not find Org system comfortable for that. I see it tries to
remember files, IDs, and agenda among various files. Not that I find
it comfortable. My way of thinking is always People or Groups, and
from there various searches are performed and that narrows drastically
the subject that has to be searched.

> My current setup has around 1200 notes in 10 files (most of them in the
> two main files, some of the notes are several pages long, but most take
> up around half a page).

People are all over the world using Org in various manners and every
day I find different ways of using Org mode.

On my side I almost never put notes in Org files. As by definition
from Wordnet, note is "brief written record". If it is brief written
record I do record it in the database under Notes related to person,
or group or opportunity or some case, or it can be related to anything
else. Then again I think of person and I can get all notes for the
person.

Org files I am using mostly for planning and project
administration. There are almost no notes, just instructions on how to
execute specific steps and there are headings with articles or
instructions that do not need execution. There are no records that are
saved for later or that do not need any execution or learning.

Org files on my side thus offer:

- hierarchical knowledge database that may be shared with other
  people, and is almost always directed to sharing with other people

- plan and project administration with tasks, whereby such subtrees
  can be shared with other people and for multiple times executed

If those are called notes by other people, alright fine. On my side
those are not just notes.

Notes I relate to objects like People, Groups, Opportunities, Cases,
so I put some notes there. But general dynamical knowledge repository
is better, that is where I mention HyperScope.

It is like database of hyperlinks that hyperlink to anything, it is
more abstract and I find that approach also versatile. No need to
define specific database for notes, all I do is defining hyperdocument
type to be "Note" and I can link it to anything else.

Semantic Synchrony
https://github.com/synchrony/smsn

Semantic Synchrony is using maybe better type of a database I do not
know, I am using SQL, SMSN uses graph database.

> Using org-rifle (https://github.com/alphapapa/org-rifle) I can
> full-text-search them with barely perceptible delay on a system
> clocked down to 1 GHz.

That is great tool for many.

Org files are for me to write complex documents like 850 kb something
like a organizational knowledge, training for each staff member,
plans, projects, tasks, etc. Majority of that stuff can remain in Org
files.

Maybe that stuff related to execution and collaboration I will move to
the database approach. All the unique ID stuff drops down forever as
database unique IDs are handling themselves without me thinking about
it, and is hard to make a mistake. It is basically putting data into
meta level. When I need a project made out of that meta data, I can
mark it all like in Dired or just mark set of such and export into Org
file and send to the person for execution.

Overall from this discussion I hope that people find some useful ways
of using Org, like org-rifle, semantic organization of stuff and
similar.

Re: One vs many directories

[Dr. Arne Babenhauserheide]

[-- Attachment #1: Type: text/plain, Size: 1881 bytes --]


Jean Louis <bugs@gnu.support> writes:

> So in general I never need to use some general search through Org
> files or any other files as my way of thinking begins with People or
> Groups and that narrows what has to be searched.

How do you deal with stuff that applies to several people?

> it comfortable. My way of thinking is always People or Groups, and
> from there various searches are performed and that narrows drastically
> the subject that has to be searched.

That does sound like it should speed up searching by directory. My mind
works differently here: I remember some concept and need to find stuff
connected to that, including people, but also additional ideas,
instructions, and just bullet points with info I might need again later
(which multiple times saved many hours of searching already).

The one thing that keeps me from that is that I often file under
specific projects, and the active projects are shifting constantly. For
that I enjoy it a lot that I only need to customize the capture
templates to add a project.

> On my side I almost never put notes in Org files. As by definition
> from Wordnet, note is "brief written record".

With note I just mean "something". Mostly its bullet points with tasks,
some links and references into source-code which allows me to quickly
take up a tasks after some downtime.

> Overall from this discussion I hope that people find some useful ways
> of using Org, like org-rifle, semantic organization of stuff and
> similar.

I hope so, too.

Thank you for describing the tools you use!

I for one am still working on my workflow, and I guess that 10 years
from now it won’t be the same as today. I hope that learning about other
ways to work with org will help me a lot in future.

Best wishes,
Arne
-- 
Unpolitisch sein
heißt politisch sein
ohne es zu merken

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 1125 bytes --]

Re: One vs many directories

[Jean Louis]

* Dr. Arne Babenhauserheide <arne_bab@web.de> [2020-11-22 01:48]:
> > So in general I never need to use some general search through Org
> > files or any other files as my way of thinking begins with People or
> > Groups and that narrows what has to be searched.
> 
> How do you deal with stuff that applies to several people?

From database viewpoint there are

- accounts (which means companies, groups, entities, like "People who
  wish to get employed in Europe")

- there are contacts, that may belong to account, additionally belong
  to company (also account), additionally be member of account, so
  there are 3 groupings for each contact how that contact may be
  related to account. If it is main account such as "Welders" or if
  maybe under "Company" is written "welders" (not quite correct) in
  reality it does not matter.

- then there are lists to which other lists belong. Account A and
  account B, C, D can belong to list 01. Various accounts can be put
  together in uniting lists. Those lists are encompassing other lists,
  not individual people but people in the list (account) usually
  unless there is only one in the account. Those lists I am using for
  mailing them or informing them by letter, SMS, etc. Geologists and
  mining engineers and metallurgists are 3 different accounts but if
  all of them speak Swahili both in Kenya and Tanzania and are in the
  related branch of economy so they can be sent same type of
  information.

Then there are groups, which is just another name for a new list. Then
there are tags. I can freely tag account, contact or anything else. By
tags I can finely select specific people belonging to specific group.

There are account types and group types.

Tags by itself have its own description or purpose to name it type.

Some people introduce other people, few of them introduced
thousands. So contacts have a column "introduced by". That becomes
very handy when talking to somebody and it also helps in awarding
introduces. It helps when people place their hyperlinks and become
automated introducers (lead generation).

When I know that person belongs to some group of people and I have to
write email and I know it is better to inform everybody, then there is
action to assign identity from which I am sending email. It could be
public or internal identity with different email address. Emails to
that person would always go from designated email address without me
thinking about it. Then there are Cc and Bcc fields and in those
fields I could designate: to inform same contact to each of email
addresses with same message, and to inform group of people each
time. Thus if writing to one contact all others get informed through
same message. But I am not thinking about who belongs to the group,
and what are their email addresses, that gets inserted
automatically. Email composition is usually inside of Emacs.

Sending files? If contact is in the group and others need to see the
file, then Cc and Bcc fields work in the same way, file sent to one
person is sent to other members of the group.

Sometimes contact tells me to please exclude some people from Cc, so I
just go into definition of identities for contact and exclude such.

SMS I am sending by using kdeconnect.el package so any SMS I send this
way it is getting recorded to the contact. If I need to send SMS to
the group, then same SMS could be sent to whole group.

If the note on one contact is related to other people then is trivial
to automate to inform other people on that particular note.

If any group of people is there, filing files under group does not
make much sense to me. I would not file it as Group/ABC/file-01.pdf

I would file it under By-ID/Contact-ID-123/Group/file-01.pdf as files
are normally sent by one email address of one person or under one
name. I deal with people not with empty groups that do not
communicate. But group is important, so there can be /Group/ directory
on the file system where all contacts of one group can be symlinked
automatically if it happens that I need to search by Group on file
system, but I don't. I search for groups in the database and see list
of contacts there and then jump to contact.

Same thing with invoices, they are written either to company or to
individual or some individual in some company. I will not file or act
based on invoice. I have to act based on authorirative or maybe
hierarchically higher object first.

In general is always good to make a list of things and then lists are
foundation for dealing easier with whatever groups of anything.

> > it comfortable. My way of thinking is always People or Groups, and
> > from there various searches are performed and that narrows drastically
> > the subject that has to be searched.
> 
> That does sound like it should speed up searching by directory.

You remember that I do not search by directory.

Computer stuff I search by directory, like
Work/Computer/Devices/Dictaphone or Work/Computer/Graphics/shadow

Stuff related to any entity, organization, group of people, mailing
list or list that encompasses other lists, I am locating in the
database but the file system location is automatically expanded from
the unique IDs. So I need not know where is exactly directory for the
group ABC, but I know if I press F8 that I am right there. Then files
are sorted for the group or account normally by date, but could be by
License number or something else. It is mixture of database relational
jump to base directory for specific account/company/group and
sub-directories. Base directory I do not need to know, those
sub-directories are few and search becomes very fast. No special
database is needed for this as one can keep hundreds of groups and
people in simple Lisp structures saved in files.

> My mind works differently here: I remember some concept and need to
> find stuff connected to that, including people, but also additional
> ideas, instructions, and just bullet points with info I might need
> again later (which multiple times saved many hours of searching
> already).

What I learn is that concept of management of information in computer
better works by the concept of associations or relations as that is
similar how mind works. I do not see much difference here in thinking
how things information should be stored.

I have some hobby and there are paper books, images, videos, DVDs,
digital books related to that hobby. All that is sorted by Authors as
that is one attribute that those things have. Additionally there are
subjects. Authors's individuals works are symlinked to various
multiple subjects as multiple subjects (tags) related to same Author's
work. If author produces only for one subject then his name is
symlinked to that subject. Sometimes there are 2 or more authors
working together, so their directory is also symlinked to individual
authors.

Similar concept applies to music or videos, it is sorted by authors
and bands, and then individual works are sorted by its type of music
or genres of videos.

Images are sorted by people who are on the image. But I do not think
of that, I just say "Adrian" and image is sorted there in appropriate
date folder for that person. Computer thinks WHERE is the location,
not me. All other files are also being sorted like that and it is work
in progress.

> The one thing that keeps me from that is that I often file under
> specific projects, and the active projects are shifting
> constantly. For that I enjoy it a lot that I only need to customize
> the capture templates to add a project.

In the dynamical knowledge repository backed by PostgreSQL I have
subtrees and each subtree has its name. I am more familiar with
structured data in the database. So if I wish to have equivalent of
capture it becomes trivial, press key, choose set and insert whatever
note or task or article, anything. It just has little different
structures.

I do not need to setup extra capture configuration as there are
already sets or subtree names. I am quickly locating appropriate
substree name and filing under.

Something I wish to file need not be Org mode, but it can be. A note
can be written in any text mode and filed in the set or
subtree. Markdown, Koutliner, it can be Perl snippet, one liner,
excerpt from email or message mode note, why not. SQL snippet, Emacs
Lisp or Scheme. Normal text. I feel more free this way. 

Hyperdocuments have its types like WWW, Gopher, Gemini, FTP, SFTP,
Note, Task, Video to be played at exact time, local file, YouTube
video, Dired directory, Program Launch, PDF, PDF by query, PDF by page
number, Emacs Lisp, Org Heading, Org file, Message ID, Set, and so
on.

These days I am thinking to make self-defining types and I mean to
place the code inside of the hyperdocument type entry that defines how
that entry should be handled. User accessing database at University
Makerere could access hyperdocument that designates that not only
video file has to be opened in the browser, but that window has to be
split so that annotations related to video file has to be shown in
same time. But user accessing such hyperdocument need not necessarily
know that its viewing or launching is customized by the type of the
hyperdocument defined somewhere else, not in software directly. It
sounds similar to Javascript launched in web browsers with full
liberty in computing. Emacs Lisp snippets would be pulled from
database directly and executed on users' computer. File transfers are
non existent and file system is not there. Nothing gets really
downloaded or saved, just loaded into memory and executed.

Just thinking of a hyperdocument type or object type "comment" or
"feedback". When activating such user loads few functions, write the
comment, functions can also say what will be done with the
comment. School teacher or head teacher acts upon it or comments are
simply published.

Sets for now do not have types, but why not. If set has type of topics
and topics allow comments there is automatic forum and discussion.

> I for one am still working on my workflow, and I guess that 10 years
> from now it won’t be the same as today. I hope that learning about other
> ways to work with org will help me a lot in future.

Same on my side.

Summary for me:

- navigation by hierarchy is one way, we all use it to search and file
  documents
  
- searching through indexed database is different way, it is not good
  for filing anything

- direct relational access where computer locates the file is third
  way for knowledge management

- it is better designing filing, sorting and retrievals by concepts of
  mind or how mind works

- any objects or pieces of information shall have its actions that act
  upon it and that can hyperpush the user to other pieces of
  information

- general increase of hyperlinking by relations helps in creation of
  better systems.

Here is how I have implemented simple versioning system that is
decided by program and not the database. It was just before hour.

(defun hyperscope-vc (table column id &optional description)
  "Simple version system."
  (let* ((value (rcd-db-get-entry table column id *hs*)) ;; takes DB value
	 (value (sql-escape-string value)) ;; prepares value for SQL
	 (description (if description (sql-escape-string description) "NULL"))
	 (sql (format "INSERT INTO vc (vc_table, 
                                      vc_column, 
                                      vc_tableid, 
                                      vc_value, 
                                      vc_description) VALUES ('%s', '%s', %s, %s, %s)
                         RETURNING vc_id"
		      table column id value description))
	 (id (rcd-sql sql *hs*)))
    (if id id nil)))

Above function takes the value from any table, any column, by ID and
inserts the value into the `vc` table. 

Then comes this function below that edits hyperdocument as Org file
with Org mode. Before updating it is using `hyperscope-vc` version
function to basically save the old value in the separate table. If I
want to see diffs I think it is also trivial to do.

(defun hlink-edit-org (id)
  (let* ((blob (hlink-description-value id))
	 (blob (if blob blob  ""))
	 (buffer-name (format "HyperScope Editing ID: %d" id))
	 (new-blob (read-from-buffer blob buffer-name 'org-mode)))
    (hyperscope-vc "hlinks" "hlinks_description" id)
    (rcd-db-update-entry "hlinks" "hlinks_description" "text" id new-blob *hs*)))

Re: One vs many directories

[briangpowell]

[-- Attachment #1: Type: text/plain, Size: 16007 bytes --]

* Strongly suggest looking into Emacs' vlf-mode and the newer vlfi-mode

** That is Very-Large-File-Mode & Very-Large-File-Improved-Mode for issues
you're experiencing & if not, simply because they're very useful &
interesting & fun Emacs Modes to explore & put into your toolbox

https://www.emacswiki.org/emacs/VLF

https://github.com/m00natic/vlfi

* You mentioned other types of GREP, I second that, and the indexing
suggestion--I remember long ago using SGREP which is Simple-GREP, used
indexing & was much faster than the usual grep implementations for some
things; but, this is at the expense of the fancier & more elaborate GREP
functions

** You mention RipGrep--thanks for that, very interesting

* Which brings me to my main suggestion to you & why:

Emacs, believe it or not, has the FASTEST ENGINE available, without
augmentation in any way, for INTERACTIVE SEARCH--since the whole engine is
designed to be optimized to search-while-editing

But for many other searches, more elaborate searches, fancier GREP
searches, it's a VERY BAD choice of ways or programs to use for searching

What I mean is, say you're editing a file, and you search for your
"ProviderBuilderFactory"

Suggest you try opening a huge file--even MULTI-GIGABYTE FILES--huge files
in Emacs VLF-Mode--Very Large File-mode {which I believe can be done as a
sub-mode to/with Org-Mode}

And you can do this fearlessly since vlf-mode works by dividing the files
up for you in the background, etc.--while you're editing--but uses the same
built-in Emacs engine, optimized for such searches

And then you type:

Control-s

And start to type the first letters of "ProviderBuilderFactory"

This will interactive-search HUGE files, very quickly, and in near "Real
Time"--since this is what Emacs (implemented in C) is optimized to do--its
optimized for initial-character-searching "as you type them"--most other
engines are NOT

IF FOR NO OTHER REASON THAN IT SOUNDS LIKE FUN! And you might use vlf-mode
for other tasks you may face in the future.

Please try it out & tell how you like it--you'll never avoid opening huge
files again is one benefit

Beyond that, suggest you look into using LEX, it's as fast as you can get
for some things too.

Everything has its niche in the *nix world--which is where GREP came from,
Bell Labs, etc.--that's the Unix philosophy, Emacs & LEX tools came from
that world & the work of thousands of contributors--suggest you try these
tools too for these issues

Lastly, say you want to search for things without opening a file, you can
still use Emacs in batch-mode, at the command line, without opening a full
emacs session



















On Sat, Nov 21, 2020 at 8:34 PM Jean Louis <bugs@gnu.support> wrote:

> * Dr. Arne Babenhauserheide <arne_bab@web.de> [2020-11-22 01:48]:
> > > So in general I never need to use some general search through Org
> > > files or any other files as my way of thinking begins with People or
> > > Groups and that narrows what has to be searched.
> >
> > How do you deal with stuff that applies to several people?
>
> From database viewpoint there are
>
> - accounts (which means companies, groups, entities, like "People who
>   wish to get employed in Europe")
>
> - there are contacts, that may belong to account, additionally belong
>   to company (also account), additionally be member of account, so
>   there are 3 groupings for each contact how that contact may be
>   related to account. If it is main account such as "Welders" or if
>   maybe under "Company" is written "welders" (not quite correct) in
>   reality it does not matter.
>
> - then there are lists to which other lists belong. Account A and
>   account B, C, D can belong to list 01. Various accounts can be put
>   together in uniting lists. Those lists are encompassing other lists,
>   not individual people but people in the list (account) usually
>   unless there is only one in the account. Those lists I am using for
>   mailing them or informing them by letter, SMS, etc. Geologists and
>   mining engineers and metallurgists are 3 different accounts but if
>   all of them speak Swahili both in Kenya and Tanzania and are in the
>   related branch of economy so they can be sent same type of
>   information.
>
> Then there are groups, which is just another name for a new list. Then
> there are tags. I can freely tag account, contact or anything else. By
> tags I can finely select specific people belonging to specific group.
>
> There are account types and group types.
>
> Tags by itself have its own description or purpose to name it type.
>
> Some people introduce other people, few of them introduced
> thousands. So contacts have a column "introduced by". That becomes
> very handy when talking to somebody and it also helps in awarding
> introduces. It helps when people place their hyperlinks and become
> automated introducers (lead generation).
>
> When I know that person belongs to some group of people and I have to
> write email and I know it is better to inform everybody, then there is
> action to assign identity from which I am sending email. It could be
> public or internal identity with different email address. Emails to
> that person would always go from designated email address without me
> thinking about it. Then there are Cc and Bcc fields and in those
> fields I could designate: to inform same contact to each of email
> addresses with same message, and to inform group of people each
> time. Thus if writing to one contact all others get informed through
> same message. But I am not thinking about who belongs to the group,
> and what are their email addresses, that gets inserted
> automatically. Email composition is usually inside of Emacs.
>
> Sending files? If contact is in the group and others need to see the
> file, then Cc and Bcc fields work in the same way, file sent to one
> person is sent to other members of the group.
>
> Sometimes contact tells me to please exclude some people from Cc, so I
> just go into definition of identities for contact and exclude such.
>
> SMS I am sending by using kdeconnect.el package so any SMS I send this
> way it is getting recorded to the contact. If I need to send SMS to
> the group, then same SMS could be sent to whole group.
>
> If the note on one contact is related to other people then is trivial
> to automate to inform other people on that particular note.
>
> If any group of people is there, filing files under group does not
> make much sense to me. I would not file it as Group/ABC/file-01.pdf
>
> I would file it under By-ID/Contact-ID-123/Group/file-01.pdf as files
> are normally sent by one email address of one person or under one
> name. I deal with people not with empty groups that do not
> communicate. But group is important, so there can be /Group/ directory
> on the file system where all contacts of one group can be symlinked
> automatically if it happens that I need to search by Group on file
> system, but I don't. I search for groups in the database and see list
> of contacts there and then jump to contact.
>
> Same thing with invoices, they are written either to company or to
> individual or some individual in some company. I will not file or act
> based on invoice. I have to act based on authorirative or maybe
> hierarchically higher object first.
>
> In general is always good to make a list of things and then lists are
> foundation for dealing easier with whatever groups of anything.
>
> > > it comfortable. My way of thinking is always People or Groups, and
> > > from there various searches are performed and that narrows drastically
> > > the subject that has to be searched.
> >
> > That does sound like it should speed up searching by directory.
>
> You remember that I do not search by directory.
>
> Computer stuff I search by directory, like
> Work/Computer/Devices/Dictaphone or Work/Computer/Graphics/shadow
>
> Stuff related to any entity, organization, group of people, mailing
> list or list that encompasses other lists, I am locating in the
> database but the file system location is automatically expanded from
> the unique IDs. So I need not know where is exactly directory for the
> group ABC, but I know if I press F8 that I am right there. Then files
> are sorted for the group or account normally by date, but could be by
> License number or something else. It is mixture of database relational
> jump to base directory for specific account/company/group and
> sub-directories. Base directory I do not need to know, those
> sub-directories are few and search becomes very fast. No special
> database is needed for this as one can keep hundreds of groups and
> people in simple Lisp structures saved in files.
>
> > My mind works differently here: I remember some concept and need to
> > find stuff connected to that, including people, but also additional
> > ideas, instructions, and just bullet points with info I might need
> > again later (which multiple times saved many hours of searching
> > already).
>
> What I learn is that concept of management of information in computer
> better works by the concept of associations or relations as that is
> similar how mind works. I do not see much difference here in thinking
> how things information should be stored.
>
> I have some hobby and there are paper books, images, videos, DVDs,
> digital books related to that hobby. All that is sorted by Authors as
> that is one attribute that those things have. Additionally there are
> subjects. Authors's individuals works are symlinked to various
> multiple subjects as multiple subjects (tags) related to same Author's
> work. If author produces only for one subject then his name is
> symlinked to that subject. Sometimes there are 2 or more authors
> working together, so their directory is also symlinked to individual
> authors.
>
> Similar concept applies to music or videos, it is sorted by authors
> and bands, and then individual works are sorted by its type of music
> or genres of videos.
>
> Images are sorted by people who are on the image. But I do not think
> of that, I just say "Adrian" and image is sorted there in appropriate
> date folder for that person. Computer thinks WHERE is the location,
> not me. All other files are also being sorted like that and it is work
> in progress.
>
> > The one thing that keeps me from that is that I often file under
> > specific projects, and the active projects are shifting
> > constantly. For that I enjoy it a lot that I only need to customize
> > the capture templates to add a project.
>
> In the dynamical knowledge repository backed by PostgreSQL I have
> subtrees and each subtree has its name. I am more familiar with
> structured data in the database. So if I wish to have equivalent of
> capture it becomes trivial, press key, choose set and insert whatever
> note or task or article, anything. It just has little different
> structures.
>
> I do not need to setup extra capture configuration as there are
> already sets or subtree names. I am quickly locating appropriate
> substree name and filing under.
>
> Something I wish to file need not be Org mode, but it can be. A note
> can be written in any text mode and filed in the set or
> subtree. Markdown, Koutliner, it can be Perl snippet, one liner,
> excerpt from email or message mode note, why not. SQL snippet, Emacs
> Lisp or Scheme. Normal text. I feel more free this way.
>
> Hyperdocuments have its types like WWW, Gopher, Gemini, FTP, SFTP,
> Note, Task, Video to be played at exact time, local file, YouTube
> video, Dired directory, Program Launch, PDF, PDF by query, PDF by page
> number, Emacs Lisp, Org Heading, Org file, Message ID, Set, and so
> on.
>
> These days I am thinking to make self-defining types and I mean to
> place the code inside of the hyperdocument type entry that defines how
> that entry should be handled. User accessing database at University
> Makerere could access hyperdocument that designates that not only
> video file has to be opened in the browser, but that window has to be
> split so that annotations related to video file has to be shown in
> same time. But user accessing such hyperdocument need not necessarily
> know that its viewing or launching is customized by the type of the
> hyperdocument defined somewhere else, not in software directly. It
> sounds similar to Javascript launched in web browsers with full
> liberty in computing. Emacs Lisp snippets would be pulled from
> database directly and executed on users' computer. File transfers are
> non existent and file system is not there. Nothing gets really
> downloaded or saved, just loaded into memory and executed.
>
> Just thinking of a hyperdocument type or object type "comment" or
> "feedback". When activating such user loads few functions, write the
> comment, functions can also say what will be done with the
> comment. School teacher or head teacher acts upon it or comments are
> simply published.
>
> Sets for now do not have types, but why not. If set has type of topics
> and topics allow comments there is automatic forum and discussion.
>
> > I for one am still working on my workflow, and I guess that 10 years
> > from now it won’t be the same as today. I hope that learning about other
> > ways to work with org will help me a lot in future.
>
> Same on my side.
>
> Summary for me:
>
> - navigation by hierarchy is one way, we all use it to search and file
>   documents
>
> - searching through indexed database is different way, it is not good
>   for filing anything
>
> - direct relational access where computer locates the file is third
>   way for knowledge management
>
> - it is better designing filing, sorting and retrievals by concepts of
>   mind or how mind works
>
> - any objects or pieces of information shall have its actions that act
>   upon it and that can hyperpush the user to other pieces of
>   information
>
> - general increase of hyperlinking by relations helps in creation of
>   better systems.
>
> Here is how I have implemented simple versioning system that is
> decided by program and not the database. It was just before hour.
>
> (defun hyperscope-vc (table column id &optional description)
>   "Simple version system."
>   (let* ((value (rcd-db-get-entry table column id *hs*)) ;; takes DB value
>          (value (sql-escape-string value)) ;; prepares value for SQL
>          (description (if description (sql-escape-string description)
> "NULL"))
>          (sql (format "INSERT INTO vc (vc_table,
>                                       vc_column,
>                                       vc_tableid,
>                                       vc_value,
>                                       vc_description) VALUES ('%s', '%s',
> %s, %s, %s)
>                          RETURNING vc_id"
>                       table column id value description))
>          (id (rcd-sql sql *hs*)))
>     (if id id nil)))
>
> Above function takes the value from any table, any column, by ID and
> inserts the value into the `vc` table.
>
> Then comes this function below that edits hyperdocument as Org file
> with Org mode. Before updating it is using `hyperscope-vc` version
> function to basically save the old value in the separate table. If I
> want to see diffs I think it is also trivial to do.
>
> (defun hlink-edit-org (id)
>   (let* ((blob (hlink-description-value id))
>          (blob (if blob blob  ""))
>          (buffer-name (format "HyperScope Editing ID: %d" id))
>          (new-blob (read-from-buffer blob buffer-name 'org-mode)))
>     (hyperscope-vc "hlinks" "hlinks_description" id)
>     (rcd-db-update-entry "hlinks" "hlinks_description" "text" id new-blob
> *hs*)))
>
>
>

[-- Attachment #2: Type: text/html, Size: 18217 bytes --]

Re: One vs many directories

[Jean Louis]

* briangpowell <briangpowellms@gmail.com> [2020-11-22 05:48]:
> Emacs, believe it or not, has the FASTEST ENGINE available, without
> augmentation in any way, for INTERACTIVE SEARCH--since the whole engine is
> designed to be optimized to search-while-editing

Interesting, I did not know about that.

> But for many other searches, more elaborate searches, fancier GREP
> searches, it's a VERY BAD choice of ways or programs to use for
> searching

Do you mean to run grep on file or buffer while editing?

Or outside of editor?

> What I mean is, say you're editing a file, and you search for your
> "ProviderBuilderFactory"
> 
> Suggest you try opening a huge file--even MULTI-GIGABYTE FILES--huge files
> in Emacs VLF-Mode--Very Large File-mode {which I believe can be done as a
> sub-mode to/with Org-Mode}
> 
> And you can do this fearlessly since vlf-mode works by dividing the files
> up for you in the background, etc.--while you're editing--but uses the same
> built-in Emacs engine, optimized for such searches
> 
> And then you type:
> 
> Control-s
> 
> And start to type the first letters of "ProviderBuilderFactory"
> 
> This will interactive-search HUGE files, very quickly, and in near "Real
> Time"--since this is what Emacs (implemented in C) is optimized to do--its
> optimized for initial-character-searching "as you type them"--most other
> engines are NOT

Does occur also uses that built-in speedy search? Would it work with
your library? Now I am researching it and I see vlf-occur

I have tried using vlf-occur in this email and what happened seems to
be error:

- M-x vlf-occur
- Tried with word "has"
- found many "has" in vlf-occur
- I press RET on the line there
- I get asked "Chunk modified, are you sure?" I say yes.
- my email buffer is erased at that moment and cursor switches to erased buffer
- then to undo stuff I press undo

So that is bug. User should never get buffer erased.

> IF FOR NO OTHER REASON THAN IT SOUNDS LIKE FUN! And you might use
> vlf-mode for other tasks you may face in the future.

I am interested to research the possibility of expanding large number
of database items into the file which would then be hyperlinked. Would
some mode like Org mode work with it? I would use Hyperbole or Org
links or goto-address-mode

The file would be one liner expansion of database entries.

By using quick vlf-occur (at this moment I just imagine it is quick,
but do not have a feeling) I would quickly locate some entries.

I would then click or activate the button to move to specific other
database entries or outside hyperlinks.

> Lastly, say you want to search for things without opening a file, you can
> still use Emacs in batch-mode, at the command line, without opening a full
> emacs session

Provide the one line to understand it.

Side thought: I remember I was making website search engines back in
time with grep only. Each HTML file was converted to one line of text,
something like:

LOCATION:::TITLE:::KEYWORDS:::DESCRIPTION:::HERE COMES LONG LINE OF BODY

Then grep was used to quickly find results which are formatted on the
website page. And yet we can just find complex software for website
searching on Internet these days.

Re: One vs many directories

[Palak Mathur]

On Fri, Nov 20, 2020 at 6:34 PM Texas Cyberthal
<texas.cyberthal@gmail.com> wrote:
>
> Having a tall directory tree with many leaves and branches is against
> Org's philosophy.
>
> Here is my argument that such a structure is objectively correct for
> personal info management:
>
> https://github.com/cyberthal/10-Bins-template
>
> For the record, Org works fine with this, although I had to do a bit
> of config to get around the default philosophy.
>

I read through the README but it seems overwhelming to have 10
directories. I am not saying it is not good just that I will not be
able to handle those. Even now when I have just three main files -
inbox, work and archive, I am very bad at refiling items (I do it but
rarely). I mostly rely on Agenda views to make things make sense for
me. With 10 directories, I will surely never be able to refile
anything as half of the time I will just be spending time in deciding
where the item belongs.

It surely will be good for more diligent people unlike me.

Re: One vs many directories

[Jean Louis]

* Palak Mathur <palakmathur@gmail.com> [2020-11-21 09:13]:
> On Fri, Nov 20, 2020 at 6:34 PM Texas Cyberthal
> <texas.cyberthal@gmail.com> wrote:
> >
> > Having a tall directory tree with many leaves and branches is against
> > Org's philosophy.
> >
> > Here is my argument that such a structure is objectively correct for
> > personal info management:
> >
> > https://github.com/cyberthal/10-Bins-template
> >
> > For the record, Org works fine with this, although I had to do a bit
> > of config to get around the default philosophy.
> >
> 
> I read through the README but it seems overwhelming to have 10
> directories. I am not saying it is not good just that I will not be
> able to handle those. Even now when I have just three main files -
> inbox, work and archive, I am very bad at refiling items (I do it but
> rarely). I mostly rely on Agenda views to make things make sense for
> me. With 10 directories, I will surely never be able to refile
> anything as half of the time I will just be spending time in deciding
> where the item belongs.
> 
> It surely will be good for more diligent people unlike me.

Thank you for nice description of your file system. In my opinion for
your use case the 10 Bins system looks very appropriate.

One point I wish to present, that is that WHERE the file is stored
need not be known to user. And when file is retrieved this also need
not be known  to user as user may search how Texas said by X methods.

Since Texas have presented 10 Bins now I am eagerly wanting to
organize all of the file system into structure that does not let me
think where the file really is located. My structure is so much more
complex but I do not think about it, computer is filing it for me. I
think of people, groups, lists and I find files by thinking, not by
looking into file system at least not so much. When I think of Texas I
write: Texas, I find 12 contacts related to Texas, then I write
"Emacs" or "Cyber" if I remember Cyber. From there on, I can see
hyperlinks to 10 Bins system, I can find emails of Texas Cyber and
find files and his Org file as well containing notes to Texas. But I
have no clear idea where that file is really located on the file
system. When finding the file I find it by X methods without knowing
where it is located on file system.

Basic structure any filing including emails could be following:

1. INBOX, items are coming here, if inbox is not empty something is
    wrong. Handled items should be filed appropriately. I file emails
    into maildir folders like ~/Maildir/person@example.com 

2. PENDING, those are things that cannot be handled for some reason,
     and because they are not handled they cannot be yet filed.
     
3. OUTBOX, those are things that need to be filed or dispatched. For
     example fax that has to be printed should be there. PDF export of
     Org file can be there as it has to be placed on USB stick to be
     brought to printer. SMS notes to be yet sent to people or files
     to be sent to people could be there. OUTBOX can be also handled
     by other people, not necessarily the organizer of things.

10 Bins system expands that not so related and offers more structure.

There is need for org-of-org.org file that is meta Org file and that
could be created on-the-fly

Meta Org file by imagination named here org-of-org.org would be
created based on information of:

- filing the files and any types of hyperlinks like videos, movies,
  etc. including Org files. Upon filing such hyperlink has its
  structure stored somewhere.

- retrieving of the files, as some files are not filed but are
  retrieved or launched. Retrieving files could then be logged or
  accounted in the file system or database for later on-the-fly
  referencing from org-of-org.org file.

Then org-of-org.org file would be created on the fly:

- it would create hierarchy of all filed files that helps the user
  locate the file easy without even knowing where the file is on file
  system

- it would use the list of retrieved files to create the structured
  hierarchy of the retrieved files by its subjects, tags, groups

Then user can click and get the meta Org file created on the fly and
pinpoint where else to go with attention.

Re: One vs many directories

[Jean Louis]

* Texas Cyberthal <texas.cyberthal@gmail.com> [2020-11-21 03:35]:
> Having a tall directory tree with many leaves and branches is against
> Org's philosophy.

Thank you for your nice ideas.

Here is your fellow classifier of information.

I don't know what is Org's strategy but if you mean many deep
subheadings my deepest is with 7 stars ******* and quite I have 7 even
of them in 850 Kb file and there are thousands of Org files on my
system.

Point is that apparently it becomes harder and hard to organize if one
does not have enhanced system of organizing. Org is rather for notes
and it not for everything

> Here is my argument that such a structure is objectively correct for
> personal info management:
> 
> https://github.com/cyberthal/10-Bins-template
> 
> For the record, Org works fine with this, although I had to do a bit
> of config to get around the default philosophy.

Your system is understandable and great idea.

Some of first ideas on filing files and notes and any piece of
knowledge in an organized manner have been presented by Doug
Engelbart, forfather of modern computing and I recommend reviewing his
work for your further thinkering.

Personally I have created similar filing system and now I may upgrade
by using parts of your offered paradigm. Maybe you pick up some of
these ideas as well.

In my life I deal with people including me and groups of people. One
person can belong to various groups of people. As both groups and
people can have same names all of them have their unique ID
numbers. And I work both with the file system and the PostgreSQL
database in background.There are just few directories that I define at
beginning, those are:

- Directories for groups or accounts how I call it since years, for
  example /home/me/data/accounts/By-ID and
  /home/me/data/accounts/By-Name

- Directories for individuals for example /home/me/data/people/By-ID
  and /home/me/data/people/By-Name

Those By-Name directories are symlinks to By-ID/UNIQUE-ID and those
symlinks need to be updated from time to time as people change their
names and there may be duplicate names, so name can look like
John-Doe-1 or John-Doe-2, those are just helpers

After that I forget anything about the above structure, so I do not
work any more with remembering the structure, not at all.

Then there are helper programs in Dired and outside of Emacs that
should work with any file manager. If file manager supports
external scripts such as Rox filer or Midnight Commander or Nautilus,
that is fine.

Let us say that I wish to file something related to the Civil War, at
first such account should exist in the database. So if it does not
exist it is created very quickly at moment of filing.

Accounts have their location in terms of the address not quite in
terms of a continent or they may be without location.

I would press s-f in Dired and choose the proper account by using
completion, normally helm. Then the file is filed under date of filing
there.

During that process I know nothing where the file is located
exactly. That is what program does for me, I am not remembering the
location neither I know where it is but I could find it when I wish.

Files goes there:

/home/me/data/accounts/By-ID/102/2020/11/2020-11-21/file-here.org and
/home/me/data/accounts/By-Name/Civil-War/2020/11/2020-11-21/file-here.org
pointing as symlink to above unique ID

That is all, after that I forget about the file.

When it comes the time to research the subject again, I just browse or
find the account "Civil War" and I say: Go to files

Then the files of Civil War are opened in front of me and I can find
it by using find-dired, I can find it by using system locate tool that
is very flexible as I can search by its specific directory, right?

Let us say I deal with person Joe Doe, normally I look into the
database for the person, and database also have location
information. There are sometimes Joe Does in various countries in
various cities so I have to quickly locate the right Joe Doe.

Once I find the Joe Doe, at a click of the key F4 I am already in the
Org file for Joe Doe. If Org file does not exist it is created on the
file and contains all the basic information I usually need to manage
for Joe Doe.

As if I am sending tasks to Joe Doe and organization X has relation to
Joe Doe, then why should I keep Joe Doe's information in one Org file?
Or few that may be personal?

Personal Org file I keep under my ID's directory. Org file related to
specific person I keep in that specific person's ID directory. My
personal Org file may point or hyperlink to some persons. Though I
find reference to such hyperlinks through communication and various
lists, and not through the list I make in my personal Org file.

I can just save the file and forget about it. But I never remember
where the file is located because that is what computer does for me. I
do not know the name of the file. That is what computer does for
me.

Another user of the computer would not know the location of file but
user would be able to always work with the file and to locate quickly
the file.

The paradigm of the PostgreSQL and general relational database entries
is reflected on the file system that becomes relational file system.

When searching in the database for anything, I am not looking where
such entry is stored, I am looking for the name or other meaning
related to the entry. Then I can find all other pieces of information
related to that entry.

Filing on file system and general filing of any pieces of information
should be related to each other as much as necessary so that by
finding one piece of information one may browse or come to those other
pieces of information through its relations.

When I search for Joe Doe, I can press F3 and one Org file is opened,
constructed on the fly, that shows me all needed relations of this
contacts:
		    ________________________________

		     PROFILE FOR CONTACT ID: 239917

			       NAME HERE
		    ________________________________


		   Saturday, November 21 2020, 08:08


Table of Contents
_________________

1. Profile for Joe Doe, contact ID: 239917
2. Pictures for contact 239917
3. Notes for contact ID: 239917
4. SMS list
5. Mailing list subscriptions
6. Mailings received
7. Interactions
8. Emails
9. Calls

In every subheading of such automatically on-the-fly created Org file
I have hyperlinks pointing to pieces of information related to that
person or actual notes, SMSs inside. Emails are in email folders that
can be quickly opened without thinking where such email folder is
located. 

Treefactor has paradigm of filing into 10 bins. I do not find it is
idiosyncratic even if it is peculiar to your system of filing. It is
so much similar to the ideas of Doug Engelbart how things should be
filed that other users can also access files by X type of queries and
that you as main user can find it easily.

> 1 Make info easily retrievable when needed. The next time you need
>  the info, you will think of it in X way. So file it at X location,
>  to convenience future you.

That is right.

By general principle to make it easily retrievable when needed I agree
on that. My searching goes by group names and by individual names and
locations and other attributes of those groups and individuals. I may
search for the country name and I am ending up in the folder related
to countries. But from countries I may not go to individuals. As if I
file a note about country then it is about legalities like various
laws, security, geology and similar. When the need arises in future I
can easily link information.

The SQL table `countries` is similar to following:

 countries_id 
 countries_name
 countries_code
 countries_prefix (phone prefix)

There are also tables `region` and `continents`, and then there are
groups of countries, for example there is group Europe and there is
group European Union and Schengen Zone.

Then if I need some particular field I can just add it to the table
`countries` such as:

 countries_apostille

to designate if country does provide international legalization by
Hague convention.

Then for reach table there shall be known what is the "name" part of
the able, in this case: countries_name, but sometimes it could be
countries_title or counties_hid (human ID) or similar. Once the setup
is made, the file system structure can be automatically derived.

All I want is to file this law into country profile, but not into
database, so I may invoke filing by country, continent, region or
city, you name it. Program is that decides WHERE it will be filed by
using those various identifiers.

In general I am not filing by location if such information is personal
or belongs to specific group. This is because that specific group
already has its location defined in the database and specific
individuals may have location in their profile and because my files
mostly related to people I interact with, they are many times sent by
those people to me.

If files are sent by specific date, such as image, then I am filing it
by choosing the date. Otherwise files are filed under the currend date
of the today under the file belonging to the group or individual.

Notes of person are filed in the database and they can also be on file
system. When opening the profile Org file I have access to all notes
in the same one view or one file (that does not exist on the file
system). From there I see all notes of the person such as notes in the
database and notes in the file system. If necessary I can save such
meta Org file or automatically update it next time.

> These rules sound complicated in theory, but they're easy in practice:

> 1 Useful stuff goes up.
> 2 Useless stuff goes down.
> 3 Put things where you'll find them.

At time of filing a piece of information it is hard to know what will
be useful in future and it is hard to know what and why is useful in
present time.

For this reason various grouping or lists or subsets are important
where one can file other piece of information.

For example there is group FOLLOW-UP where I put people I have to
follow-up until they are not followed up.Person could be member of the
group FRIENDS but not a friend that I follow up with communication or
bring through certain stage or process. Some people are in the group
of AGENTS and some have been sent BIRTHDAY-CARD, some are relatives of
person ABC and so on, all those relations once established should then
be quickly accessed from one piece of information to other.

My ranking is  thus based on various groups that designate ranking.

There is basic and on the fly created group of last contacts.

M-x cf-last-contacts

shows me all the last 200-300 last contacts entered into the
database by their group, country, by the date they are entered.

> 0-Inbox is first because its existence is a red flag. If it exists,
> then you should stop and refile it before anything else. Otherwise,
> arbitrary headings can be lost indefinitely in bypassed inboxes
> buried in the tree!

Good concept.

My inbox for files is usually $HOME directory or $HOME/TO-SORT or
$HOME/TO-DO

Those files need to be filed and notes created for future, or maybe
deleted. When filed they get its use, become useful.

Example is the mining license pertaining to specific person, if not
filed under that person due to nature that I am looking for people and
not files, I would not find the file that is not filed. When I speak
next time to that person I would not know what the person speaks
about:

"Hey, how do you find the licensed location?"

- now if I know the name or email of person, I can quickly search and
  find all notes, files and the license. I can be able to answer
  quickly. When dealing with thousands of people that is the way to
  go. I will know what to answer if I have filed people, groups, files
  in proper place. If I have not, I am lost and relation may get
  broken because I am not following up and not knowing what person is
  talking about. And that is business that I am losing if I am not
  organized.

> 1-Outbox is next because it is effectively the Inbox for another
> repo. A similar danger applies, although at least you know the info
> doesn't belong in the current repo.

I support this idea though personally I have not get other
repositories. I have external hard disks with videos that I cannot
possibly keep internally and I have external hosting space with
files. They are (should be) all sorted through the dynamic knowledge
repository that is named HyperScope.

If the file is under subject "Hobby" at location of external disk ABC,
then such is indexed in HyperScope. All I need to do is to click on
the file. If disk is not connected, program has to tell me where to
find the disk with the specific designation. If disk is connected and
not mounted it has to get mounted when I need it and file is opened.

All that time I need not know exactly where such file is located. All
I know is either AUTHOR name or SUBJECT of video or VIDEO FILE NAME
that I see in front of me, maybe I can see description and some tags
or marks. But where the file is really located I do not need to see.

So I spare all the time to search for files by browsing the file
system. I search for author or name and files are found
automatically.

> 2-Codex is next. Its content is terse and dense. You don't want your
> code snippets buried in your prose.

Idea is good. Personally not so important.

My software is filed under some directory and various subjects.

If it belongs to person it may be filed under specific person. Let us
say there is group: Emacs Package Authors, it is quite easy to index
authors, to file their software into their place which I do not
consciously need to know where it is, to extract the README or package
commentary and form an index. Then by using HyperScope I can quickly
access the index or update their files let us say by using `git
pull`. Or I could automate the process.

In software world authors are almost invisible as their names are
buried, maybe that is customarily I do not know but not that I agree
to that fact. I do like to know the author's name at least as there is
quite a difference if software is written by Thierry Volpiatto or by
`jtbm37`. The difference is related to trust, quality and familiarity
of software. Especially free software is used often by relation to the
community or by some principles of software creation. Those principles
originate from developers and their philosophies such as OpenBSD or
DragonflyBSD principles or philosophies, or GNU and Richard Stallman's
philosophy, or freedom pushing GNU Hyperbola/Linux-libre and GNU Guix
operating systems.

> Info also varies by retrievability. The more costly the retrieval,
> the less profitable the info.  Maintenance cost also reduces info
> profitability.

Not that I can relate to this. Retrieval is not for me important
attribute to note.

> Stable info has high retrievability and low maintenance cost. The next
> four directories are ranked primarily by stability.

> 4-Time is first, since both money and time are precisely and
> objectively quantifiable. Both are precious.

That is one necessary factor for classifications. I am using it and it
is done automatically by the date of filing or semi-automatically when
I wish to designate specific date to the filing.

> 5-Location is next, because geography is quite stable and
> fundamental, as is architecture. Both are a bit more ambiguous and
> dynamic than time and money, though.

As locations are related to people and accounts (entities, groups,
lists) my search for file under specific location goes over account or
group, not over location to account, but that is also possible.

I am searching for the group "Transport" with country "Uganda", so I
find transporters in Uganda or in city Kampala and their files are
quickly opened without me knowing where the files are, and their
emails are opened without me seeing their email address, it is just by
click as things are hyperlinked.

While I do agree to file by location, I think it is not manageable
without using the database. There are continents, 248 countries in my
database system, there are so many regions, cities, etc. My people are
located in 229 countries. Do I really want their data located under
countries? I do not think so. That would be too much geographical
information on the file system. I can search for country and find
peple in that country or that city, but not that I wish to have too
many geographical attributes on the file system as that is repetition
not necessary. My database of people has 196159 people. Majority of
them have their location, it is mostly country, often state and
city.

I can find a mining engineer in country Senegal if I wish so, that has
some work experience and I can see files pertaining to this
person. But not that I would make file system directory Senegal to
find the files for this person, as Senegal as attribute is already
related to specific subset of people in the database.

While I am using similar system of searching by location, I am not
filing necessarily by location. Unless that filing relates to the
location itself as a group such as USA/Colorado/Laws/LLC-Law.txt

Same for objects.

> 7-Name is next. Names are more ephemeral than objects. Your personal
> nomenclature evolves gradually.

Quite clear.

> The last two directories follow the priority-density rule again.

> 8-Action is next. Plans are more ephemeral than names, but more
> important than background info.

> 9-Background is last. It contains huge quantities of
> poorly-retrievable info, most of it easily replaceable.

I have plans in the database. There are plans as stragegical plans,
tactical plans, projects that fulfill specific plan step, tasks that
fullfil specific project's step, goals, purposes, etc.

When person belongs to project, person is related to that project.

If I then wish to look into the plan or project, I can also see which
people are related to plan or project and act upon it.

Set of hyperlinks in HyperScope can provide me a hyperlink to the file
where by such hyperlink is related to specific contact 123 and
specific project 234.

Hyperlinks belonging to one subset of things are then organized
together.

HyperScope is my main dynamic knowledge repository of many things,
various hyperlinks.

It can create dynamic on the fly meta Org file that points out to all
the other Org files, any kind of files as hyperlinks (not only WWW),
PDFs, movies, images, notes, people and groups, etc.

> This improves iterational velocity, accuracy, creativity, and
> effective intelligence.

While I have expressed some personal preferences and differences, I am
in full agreement that systems of knowledge organizing such as
Treefactor are useful and helpful.

And I think that every person is able to develop such for oneself. It
is not that hard or complex. It is just a paradigm that one wish and
wants to adopt and apply in life.

When used for the benefit of the group such paradigm increases the
collective IQ.

The system of organizing knowledge you have described would be so much
more useful if it would be practically used by people of the same
group. That is where it increases collective IQ tremendously.

References:

Doug Engelbart Institute - Boosting mankind's capability for coping with complex, urgent problems
https://www.dougengelbart.org/

Draft OHS-Project Plan
https://web.archive.org/web/20070704185333/http://www.bootstrap.org/augdocs/bi-2120.html

TECHNOLOGY TEMPLATE PROJECT OHS Framework 
https://www.dougengelbart.org/content/view/110/460/

Addressing Hyperdocument Content And Structure
https://www.dougengelbart.org/content/view/110/460/#2b1

Toward High-Performance Organizations A Strategic Role for Groupware 
https://www.dougengelbart.org/content/view/116/#7j

About Dynamic Knowledge Repositories (DKR)
https://www.dougengelbart.org/content/view/190/163/

Re: One vs many directories

[Texas Cyberthal]

***** Hi Ihor Radchenko,

> I am wondering what you mean by Org's philosophy. Why would it have anything to do with directories?

Org's philosophy is to have one or a handful of directories without
nesting of directories.  Users are not expected to have their Org
files in a deeply nested tree.  Org also prefers big files with large
trees rather than lots of little files.

By philosophy, I mean the dev consensus on the correct way to do
things, and coded configuration and usability biases.

I know this is Org's philosophy because I violated it thoroughly when
writing Treefactor documentation, and was told as much.  I can see how
it wouldn't be obvious to casual users.

Good idea, I'll comment on Voit's article, thanks.

***** Hi Palak Mathur,

> it seems overwhelming to have 10 directories. I am not saying it is not good just that I will not be able to handle those.

I didn't anticipate this problem.  Maybe practicing with Treefactor
and Dired would build this muscle over time.

The rules are written to be straightforward at filing time.  One need
only consider one object at a time.  Cascade filing means one need
only compare the object with one directory at a time.  The first match
wins.  I should emphasize that in the docs.

Having all your headings jumbled into three huge files sounds like a
state of permanent intractable overwhelm to me.

***** Hi Jean Louis,

You are using Hyperscope as your primary PIM but integrating it with
Org, and it sounds like you're using PostGreSQL and the directory tree
together somehow.  I don't fully understand it.

Clearly a database can do more than a directory tree alone.  The cost
is that a database is more complex to use and maintain.  So that which
can be handled by directory tree alone, should be.

> I can find a mining engineer in country Senegal if I wish so, that has some work experience and I can see files pertaining to this person. But not that I would make file system directory Senegal to find the files for this person

Of course not.  You would find a person under his name, not his
country.  The person can move to a different country, after all.  At
most you might link him to the country, as part of a list of people
from X country.  But that is something better handled by a real
database.

To clarify, Treefactor is just an Emacs package with some minimal
opinions.  10 Bins is an opinionated directory tree template.  Neither
requires the other, but they're both part of Cyborganize, my overall
PIM and publishing system.

Re: One vs many directories

[Jean Louis]

* Texas Cyberthal <texas.cyberthal@gmail.com> [2020-11-21 10:19]:
  :PROPERTIES:
  :CREATED:  [2020-11-21 Sat 12:53]
  :ID:       913ca9e4-17b4-4366-9a35-45838cace538
  :END:
> ***** Hi Ihor Radchenko,
> 
> > I am wondering what you mean by Org's philosophy. Why would it have anything to do with directories?
> 
> Org's philosophy is to have one or a handful of directories without
> nesting of directories.  Users are not expected to have their Org
> files in a deeply nested tree.  Org also prefers big files with large
> trees rather than lots of little files.

Is it? I did not know that. But it is definitely tedious to keep all
Org files in one directory or few sub-directories. That is where 10
Bins paradigm becomes useful or other similar filing software. It is
just good to file it into fixed place with fixed hyperlink that is
easily captured somewhere.

What I mean is when file is filed into

ABC/STRUCT/XYZ/001/2020/11/20/2020-11-20/

is that such link can quickly be captured in some main file like:

* ABC
  :PROPERTIES:
  :CREATED:  [2020-11-21 Sat 12:53]
  :ID:       8459366d-e95e-4a2f-9eac-c093375d733c
  :END:
** STRUCT
   :PROPERTIES:
   :CREATED:  [2020-11-21 Sat 12:53]
   :ID:       5d3b208f-dfec-4916-af7e-be784641d3f8
   :END:
*** XYZ
    :PROPERTIES:
    :CREATED:  [2020-11-21 Sat 12:53]
    :ID:       19705797-19b5-46d4-ab7d-2414589a5e68
    :END:
**** 001
     :PROPERTIES:
     :CREATED:  [2020-11-21 Sat 12:53]
     :ID:       ddf01399-3b6a-40d5-9a8c-55cadd2d08b2
     :END:
***** 2020
      :PROPERTIES:
      :CREATED:  [2020-11-21 Sat 12:53]
      :ID:       5faef432-0abc-4953-bd81-2ffae69ecec1
      :END:

[[file:ABC/STRUCT/XYZ/001/2020/11/20/2020-11-20/file.org][File.org is filed file]]

When such capture is done automatically upon filing then the link may
be reused in other files. By using the Meta Org File user
automatically creates an index of filed files and can search for the
file in the Org file itself and open the file from the Meta Org File
without knowing where the file is really located.

> The rules are written to be straightforward at filing time.  One need
> only consider one object at a time.  Cascade filing means one need
> only compare the object with one directory at a time.  The first match
> wins.  I should emphasize that in the docs.
> 
> Having all your headings jumbled into three huge files sounds like a
> state of permanent intractable overwhelm to me.

That overwhelm I felt myself and found similar way like you Texas to
file the files and I am still expanding into meta-level of organizing
things.

> You are using Hyperscope as your primary PIM but integrating it with
> Org, and it sounds like you're using PostGreSQL and the directory tree
> together somehow.  I don't fully understand it.

My primary PIM is on higher level than HyperScope as HyperScope is
part of PIM which I call CRM or Customer Relationship Management as it
does not manage only my personal information. In the practical sense
it is more or less same thing only that allows work in group. It is
just so similar to various other CRM software around.

HyperScope is not file system. It is dynamical knowledge repository of
hyperdocuments, any kinds of documents. Other similar projects are:

Semantic Synchrony
https://github.com/synchrony/smsn

Hypothes.is Annotate the web, with anyone, anywhere.
https://web.hypothes.is/

GNOWSYS: A Kernel for Semantic Computing!
https://www.gnu.org/software/gnowsys

gstudio
https://gitlab.com/gnowledge/gstudio

Project Xanadu
https://en.wikipedia.org/wiki/Project_Xanadu

Those links I am quickly inserting into buffer exactly by using
Hyperscope. I am just pressing one key W and hyperlink from HyperScope
buffer is creating those hyperlinks above.

In general it is a tree based database that hyperlinks to all kinds of
hyperdocuments. Hyperdocument is also a note in the database
itself. But it can be Org file or any other file regardless where it
is located, be it on the WWW or file system. It can be video that I
have to watch or specific part of video.

Then subtrees or the whole HyperScope database can be expanded into
one Meta Org File if I wish. It is not "export" in the sense but one
could say it is similar to export.

> Clearly a database can do more than a directory tree alone.  The cost
> is that a database is more complex to use and maintain.  So that which
> can be handled by directory tree alone, should be.

File system is database. What I keep in the PostgreSQL database it can
be kept directly in the file system while program can manage various
other relations. There is also EIEIO in Emacs which I found recently
that also may be used as a database and there is GNU Emacs database.

EDB or GNU Emacs database
https://www.gnuvola.org/software/edb/

and various other interfaces to databases.

Database need not be complex. It carries only the structure. I find
it much easier and less complex to define the database table then to
work with program that has to replicate all those algorithms that
databases have built-in.

Database is like storage for custom variables. You have already
defined 10 Bins, the database in that sense could just define more
complexities while program can handle such complexities automatically.

Database does not store files for file system, that is not my
preferred way. It just keeps the structure for filing. Same structure
could be kept with EIEIO or even with defcustom or simple lists. But
when it becomes more complex that is where database becomes useful.

I can keep 100 or 500 people in lists, but searching, indexing, unique
IDs, all those stuff can be handled by the database without me
thinking about it.

Example is with unique Org IDs in the package `org-id`. I do use such
unique IDs. But I have to have the package, somebody has to program
the package and those unique IDs may not be so unique. For example if
I copy the substree I need to take care to change the unique IDs to
something else. If Org file is copied I need to take care that unique
IDs get updated. There is human error involved.

Database can have defined field such as `TABLE_ID SERIAL NOT NULL
PRIMARY KEY` and that is is where I stop thinking about unique ID. It
is unique and it is enforced over all table of items.

If I wish to delete item, fine. If I wish to copy item from one
subtree to other, there will be new unique ID, if I just change the
parent of the item, the unique ID still remains there.

Org development does try to accommodate this principle but it is text
and files are everywhere so I see Org programs try to track the files,
some are not tracked and so on, at some point of time it becomes not
manageable.

In general beyond thinking how a table should be defined there is
nothing so much complex with databases. And that design and planning
is usually programmed so users do not even think about database and
tables.

Your link `template' on the page:
https://treefactor-docs.nfshost.com/

under:

"Here is a template you can use to start your own Textmind. "

is not working, it is asking me to sign up on the third site.

These are all very useful and logical options:

https://treefactor-docs.nfshost.com/2-commands/1-throw/

Myself I write Org files for planning and projects which are then
executed by printing them and carrying such projects in the field. In
that sense the TODO items are always repeated and I do not refile
things, almost never if ever I did so. I do not remember.

Often there are tasks related to people so each person has its Org
file here, I do not mix information pertaining to various people in
Org files. This way I can send whole file to the person for review and
modifications. RCS versioning is enough to keep things simple and to
see differences between the files.

One subject is kept in one Org file. If something is done and not
necessary to be visible it can be archived. More than that I do not
re-file things from one file to other as I simply keep right Org file
for right thing.

Concept of refiling in Treefactor is fine concept regardless of my
personal preferences.

> opinions.  10 Bins is an opinionated directory tree template.

You gave me idea to expand my personal organization into file system
that I can forget about it. 

> Neither requires the other, but they're both part of Cyborganize, my
> overall PIM and publishing system.

Definitely I need to read more about it on your website.


Jean

Re: One vs many directories

[Tim Cross]

Jean Louis <bugs@gnu.support> writes:

> My primary PIM is on higher level than HyperScope as HyperScope is
> part of PIM which I call CRM or Customer Relationship Management as it
> does not manage only my personal information. In the practical sense
> it is more or less same thing only that allows work in group. It is
> just so similar to various other CRM software around.
>
> HyperScope is not file system. It is dynamical knowledge repository of
> hyperdocuments, any kinds of documents. Other similar projects are:
>
> Semantic Synchrony
> https://github.com/synchrony/smsn
>
> Hypothes.is Annotate the web, with anyone, anywhere.
> https://web.hypothes.is/
>
> GNOWSYS: A Kernel for Semantic Computing!
> https://www.gnu.org/software/gnowsys
>
> gstudio
> https://gitlab.com/gnowledge/gstudio
>
> Project Xanadu
> https://en.wikipedia.org/wiki/Project_Xanadu
>
> Those links I am quickly inserting into buffer exactly by using
> Hyperscope. I am just pressing one key W and hyperlink from HyperScope
> buffer is creating those hyperlinks above.
>

I have used similar tools in the past. However, what I find frustrating
about them is that your now dependent on another bit of technology - a
database of some type with all the issues that adds - installation,
upgrades, maintenance, backups etc. The thing I like best about Org is
that in the end, it is just a collection of plain text documents.

I haven't had the challenges mentioned by others. My org files are
broken into directories and I have a directory hierarchy and I use
things like org id and other add ons to extend. I also have started
playing around with org-roam, which I find to be quite interesting -
especially the biity it has to handle bidirectional links etc.

I also wanted to mention another Emacs project which is quite
interesting and has actually been around a lot longer than org,
Hyperbole. I've not got a URL handy, but I'm sure you can find it with
google.

It is an interesting system which pretty much makes everything possible
in a document a hyperlink. Provides some very interesting ways of
linking documents.

My preference has always been to 'do my own thing'. I tend to look at
other information management approaches and cherry pick the bits which I
like and then replicate them in org. I don't find org as limiting as
others seem to, but I'm also quite happy to add in my own elisp to tweak
it the way I want it to be - thats why I love emacs.
--
Tim Cross

Re: One vs many directories

[Jean Louis]

* Tim Cross <theophilusx@gmail.com> [2020-11-21 13:15]:
> I have used similar tools in the past. However, what I find frustrating
> about them is that your now dependent on another bit of technology - a
> database of some type with all the issues that adds - installation,
> upgrades, maintenance, backups etc. The thing I like best about Org is
> that in the end, it is just a collection of plain text documents.

Thank you, I understand your opinion. Let me share my viewpoint.

Installing database such as PostgreSQL and configuring it is way more
simpler than installing Emacs and configuring Org mode or Emacs
itself.

It goes like this:

$ PACKAGE-MANAGER install DATABASE

Later you just do:

$ createdb MY-DB

and then insert tables, usually program does that for you.

- installation is simpler than Emacs, definitely, unless you install
  from sources which is in turn about the same as Emacs

- upgrade is not necessary in general, majority of users need not ever
  upgrade. GNU/Linux distributions make upgrades easy and flowless
  normally.

- for maintenance of database I do not know what this refers to. To me
  it means working with the database. Inserting, updating, editing
  records. That is writing text. I write text in Org and I write text
  in the database, it is about same thing without file on file
  system. It lacks maybe versioning system which in turn can be
  implemented easily to simple first backup the field into versioning
  table before editing such. Simple.

- backup of the database is way easier than backup of the file
  system.

export POSTGRESQL_BACKUPFILE="/home/data1/protected/Security/Backup/Database/PostgreSQL/`/bin/date -Iminutes`.sql.gz"

alias backupPgDB='nice -n 19 /usr/local/bin/pg_dumpall | gzip > $POSTGRESQL_BACKUPFILE'

That is all what is needed for the backup in my example. It can run by
cron job that I do not even think about it. I can send database file
by email or encrypt it by GnuPG and upload to some remote server
automatically.

> I also wanted to mention another Emacs project which is quite
> interesting and has actually been around a lot longer than org,
> Hyperbole. I've not got a URL handy, but I'm sure you can find it with
> google.

GNU Hyperbole
https://www.gnu.org/s/hyperbole

I am using Hyperbole everyday and it integrates with Org mode and can
use Org links. It is on meta-level in comparison to hyperlinking in
Org mode and in itself it is hypertextual information management
system but is only in one part comparable to Org. It is not a mode for
editing files, but that one part is named Koutliner and that is
outline mode comparable to Org with one major difference that
Koutliner provides specific ID for each part of the nodes in the
substree.

People often mix GNU Hyperbole and Org, but they are not comparable as
Hyperbole is Hyper and Org is Org. Being Hyper it works in every file
or every directory. I just wish to find out how to create buttons on
the fly to interact between Hyperscope and Hyperbole. 

> It is an interesting system which pretty much makes everything
> possible in a document a hyperlink. Provides some very interesting
> ways of linking documents.

Yes. When I enter my project directory there I keep ./HYPB directory
button file. Then there are some tasks which are easily invoked by
Hyperbole that relate to various maintenances:

General actions
===============

- <(Record the Developed Traffic incidents)>
- <(Create General Log entry)>

Search
======

Database actions
================

- <(CF: Last 200 contacts)>
- <(CF: People with mining lands)>
- <(CF: Create contact and edit it)>

Database settings
=================

- <(Turn off speech)>

Database Maintenance
====================

- <(CF: Find largest contact names)>
- <(CF: Find possible doctors)>
- <(CF: Normalize public email addresses)>
- <(CF: Normalize phone numbers)>

Now I am sure that same can be done with Org file as it offers many
kinds of hyperlinking. While Org hyperlinks can be placed in any file
and used in any mode they look ugly in non-Org text files without Org
links mode enabled.

> My preference has always been to 'do my own thing'.

My preference for other people is same, that each should find their
way and use the tools and paradigms they find familiar or appropriate.

> I tend to look at other information management approaches and cherry
> pick the bits which I like and then replicate them in org. I don't
> find org as limiting as others seem to, but I'm also quite happy to
> add in my own elisp to tweak it the way I want it to be - thats why
> I love emacs.

Of course. Me too. And I develop my own. I like to integrate
things. For example tasks in Org mode I have to delegate to people so
I click a button, choose person, choose valid email if there are many
and task is sent to person. Person gives me feedback and I update the
task conducted by that person. That is integrating.

Maybe I am in the Org file for the person, how do I call quickly this
person? If there is #+MACRO: contact-id 239917 then I can just click a
key and send SMS or initiate a call straight from the object related
to that contact in this case Org file. And it should work for any
object any file related to person or other object.

If I inspect country database, maybe contacts are related to country,
from country I should be able to find contacts of that country. From
any file or location in directories related to person ABC, I should be
able to initiate communication with such user without again finding
user's n

I am in process of transition to HyperScope as dynamic knowledge
repository.

As Org files are too many. Some are important, some not, some carry
this and that information, it is everywhere, not any more simple.

I am sorting it automatically into people's directories. Contracts are
written by Org and such contracts are related to people, they shall be
sorted by relations.

This applies in general to any kind of files. When files are sorted
they should be in the same time, same second, also indexed. Automated
indexing should also help the user. I do not mean indexed as full
file, just as meta information or hyperlink to which one can find a
quick reference.

Re: One vs many directories

[Texas Cyberthal]

Hi Jean,

> By using the Meta Org File user automatically creates an index of filed files and can search for the file in the Org file itself and open the file from the Meta Org File without knowing where the file is really located.

Such a set of links could easily grow out of date if paths change, and
I wouldn't want to maintain it.  If the paths never change, then I
would memorize the paths and linking them would be slower than Dired
walking to them.

I do have a method called "Zinks" for managing UID links.  It permits
paths to change without breaking anything, both target and source.
However, in the vast majority of cases I find it much easier to just
walk the directory tree.  I doubt one can appreciate how useful a tree
synced to one's mind is until one has experienced it.  The tree adapts
to the mind and vice versa.

There's no need to know the exact locations of files; walking there is
informative and useful.  Or, for the trivial paths, walking is so
quick that it is faster than searching.  Search spawns distracting
mismatches to read, whereas walking the tree progressively narrows
scope in a mentally comfortable way that focuses the mind while
error-checking each step.  It's very comfortable to reach the
destination and be confident from the process that I'm in the right
place.

> File system is database.

Barely.  Databaseness is a gradient with file system at one end and
PostGres at the other.

Plain text and file system are the computing foundation.  The largest
and best set of tools apply.  Departing from them loses much.

A filesystem is extremely ergonomic with the right tools, and handles
bulk data very well.  Any database, even Org's, has higher overhead.

I do intend to integrate databases into Cyborganize with Dbmind, but
have barely thought about it yet.  Cyborganize should run fine without
any database, but of course database is extremely useful for business
etc.

I just don't think paths need to be input into the database.  The
strength of the database is freedom from the file system.  It should
focus on the things a file system can't do.  For example, querying all
the people who work at X company, or who live in Y country.

Duplicate Org IDs aren't a problem in my experience.  Noticing their
existence is a good way to reconcile the split after the dust of
execution has settled.  An Org workflow shouldn't generate lots of
duplicate links.  One that does probably indicates overuse of both
links and heading duplication.  If one really does need lots of unique
IDs, it's probably a sign to move to a heavier database than Org.

I'll fix that link.  The correct URL is
https://github.com/cyberthal/Textmind-template

Glad to hear you're finding Cyborganize components useful.

Re: One vs many directories

[Jean Louis]

* Texas Cyberthal <texas.cyberthal@gmail.com> [2020-11-21 17:45]:
> Hi Jean,
> 
> > By using the Meta Org File user automatically creates an index of
> > filed files and can search for the file in the Org file itself and
> > open the file from the Meta Org File without knowing where the
> > file is really located.

> Such a set of links could easily grow out of date if paths change, and
> I wouldn't want to maintain it.  If the paths never change, then I
> would memorize the paths and linking them would be slower than Dired
> walking to them.

By principles of Doug Engelbart, once file is filed, it should be
there and nowhere else, it becomes more or less static. Then you have
some kind of DKR or dynamic knowledge repository that tracks where
such files are. Let us say that DKR provides you with the hyperlink
"K12A". This link is is then referenced from all other files.

If the underlying location of the file changes, but it should not, all
the hyperlinks remain always the same.

> I do have a method called "Zinks" for managing UID links.  It permits
> paths to change without breaking anything, both target and source.
> However, in the vast majority of cases I find it much easier to just
> walk the directory tree.  I doubt one can appreciate how useful a tree
> synced to one's mind is until one has experienced it.  The tree adapts
> to the mind and vice versa.

That is good and useful.

While I do order things in a tree it is not that I think of those
tree in hierarchical sense. By habit I could maybe remember but
because it is not necessary to remember, it is not useful, then I
don't.

What I remember is the sub-tree name. For example Emacs or Emacs Lisp,
then I just do the helm or ivy or other completion and find the
subtree. From there on I browse for information, file it, open up or
launch hyperlinks. It is not browsing the tree, it is more direct jump
to the relevant part of the three.

And if hyperlinks are represented as (hyperscope 123) then regardless
if such link moves from subtree to subtree or underlying information
changes somehow the reference to that link remains always same (as
long as ID is always same).

> There's no need to know the exact locations of files; walking there
> is informative and useful.  Or, for the trivial paths, walking is so
> quick that it is faster than searching.

That is good and isn't it general way of sorting things? I guess that
general computer users may not be aware that they could make nice
hierarchical tree of directories. System that users are offered are
file managers and such are way to abstract for today's users. Back in
time we did not have that many files, today we have loads and loads of
files. Those programes named file manager do not manage anything by
themselves. And they should.

User should choose to file a file, and by any type of paradigm, user
would maybe answer just 1-3 questions and file would be filed into
appropriate place. It could be retrieved as fast as filed. That would
be semi-automatic file management.

If all files would have its meta-data then automatic file managemet
could be possible. Then it would be known who is author, what is
subject or tag of the file, what is its title, date of writing it,
permissions (not file permissions but access permissions) and
similar. Then program could file those files automatically.

For images named IMG_2020111012345.jpg is possible to parse image
names and if not image names then the built-in EXIF data and file all
images by their dates automatically. Bunch of images coming from
camera and they need to be sorted by date so in that case it can be
automatic.

Text files do not have its meta data. Org files could be said to have
as there is TITLE and AUTHOR, DATE. But there is no rule for meta
data. Good set of principles or rules when creating files could create
meta data and by meta data files could be automatically filed and
easily retrieved.

> Search spawns distracting mismatches to read, whereas walking the
> tree progressively narrows scope in a mentally comfortable way that
> focuses the mind while error-checking each step.  It's very
> comfortable to reach the destination and be confident from the
> process that I'm in the right place.

That is exactly same mental concept that I use. It becomes pleasure as
well. I could watch videos of other similar creators as you and I can
feel the pleasure in their voices and I can understand the piece of
mind when files are sorted. Unsorted files do bring mental mess in the
person's mind.

> > File system is database.
> 
> Barely.  Databaseness is a gradient with file system at one end and
> PostGres at the other.
> 
> Plain text and file system are the computing foundation.  The largest
> and best set of tools apply.  Departing from them loses much.

I know in computing we say "database" for those more sophisticated
databases. By definition from Wordnet:

* Overview of noun database

The noun database has 1 sense (no senses from tagged texts)
1. database -- (an organized body of related information)

Thus database can be in any form. It need not be special so much
special software. It need not be software and digital data at all, it
can be paper database.

Simple text file can be excellent database for contacts for as long as
editor provides search capabilities and there is some kind of
organized way of putting data inside.

> I do intend to integrate databases into Cyborganize with Dbmind, but
> have barely thought about it yet.  Cyborganize should run fine without
> any database, but of course database is extremely useful for business
> etc.

Maybe complex database is not needed. As I said there are various
approaches and it can be all simple.

A database can be also simple LISP data structure, it could be list of
hashes securely saved on the file system with possibility to add,
modify, delete records. Multiple databases can be made this way for
multiple subjects, let us say PEOPLE, GROUPS, LISTS, etc. Then
underlying file system can follow the defined structure.

> I just don't think paths need to be input into the database.  The
> strength of the database is freedom from the file system.  It should
> focus on the things a file system can't do.  For example, querying all
> the people who work at X company, or who live in Y country.

File system can be used for the same:

- symlink `People/Joe Doe` to Countries/USA/FL/Miami
- symlink `People/Joe Doe` to Groups/Emacs Users/
- symlink `People/Joe Doe` to Companies/USA/PROGRAMMING INC/

To find the information:

locate -A "Joe Doe" PROGRAMMING

or

locate -A "Joe Doe" Miami

or just

locate FL/Miami

By symlinking into various attributes you can get selections easily.

> Duplicate Org IDs aren't a problem in my experience.  Noticing their
> existence is a good way to reconcile the split after the dust of
> execution has settled.  An Org workflow shouldn't generate lots of
> duplicate links.

It should not but it does.

Projects are repeatable sometimes. Need not be single person
projects. Project that I do may have unique IDs and I have to assign
same project to ABC number of people. Then I copy the subtree into the
file of that person. Unique IDs are also copied. Of course it is
manageable if I create some functions to delete and regenerate unique
IDs.

What is handy with unique IDs is that emails and threads with tasks
assigned by email, containing the unique ID, can easily be located.

> One that does probably indicates overuse of both links and heading
> duplication.  If one really does need lots of unique IDs, it's
> probably a sign to move to a heavier database than Org.
> 
> I'll fix that link.  The correct URL is
> https://github.com/cyberthal/Textmind-template

I have cloned it and I see it is something familiar to you, not that I
can understand your mind flow. In general I find any such system of
organization useful for people to learn from, to adopt it or develop
their way of thinking and managing files.

Re: One vs many directories

[Ihor Radchenko]

>> I am wondering what you mean by Org's philosophy. Why would it have anything to do with directories?
>
> Org's philosophy is to have one or a handful of directories without
> nesting of directories.  Users are not expected to have their Org
> files in a deeply nested tree.  Org also prefers big files with large
> trees rather than lots of little files.
>
> By philosophy, I mean the dev consensus on the correct way to do
> things, and coded configuration and usability biases.

I believe that org support all possibilities. The user can decide to
keep many (possibly nested) org files, a few large org files, or
anywhere in between. There are several parallel feature sets allowing to
work in a single file as well as with a bunch of smaller files.

For a single file, the user can search headings with org-goto (without a
need to explicitly travel through all the nesting headline levels),
reveal only headings satisfying certain keyword/tag/any other search
criteria with org-sparse-tree, or built agenda views restricted to a
single file (or even subtree).

For multiple files located anywhere in the filesystem, there is always
org-refile capable of filing the information to proper place or
searching deeply nested headlines with ease regardless of the file the
information is physically located in. Headlines from multiple files can
be grouped using agenda views for any given search criteria (showing
todo items or items for a single day/week is just a tiny subset of what
agenda can do).

Best,
Ihor

Texas Cyberthal <texas.cyberthal@gmail.com> writes:

> ***** Hi Ihor Radchenko,
>
>> I am wondering what you mean by Org's philosophy. Why would it have anything to do with directories?
>
> Org's philosophy is to have one or a handful of directories without
> nesting of directories.  Users are not expected to have their Org
> files in a deeply nested tree.  Org also prefers big files with large
> trees rather than lots of little files.
>
> By philosophy, I mean the dev consensus on the correct way to do
> things, and coded configuration and usability biases.
>
> I know this is Org's philosophy because I violated it thoroughly when
> writing Treefactor documentation, and was told as much.  I can see how
> it wouldn't be obvious to casual users.
>
> Good idea, I'll comment on Voit's article, thanks.
>
> ***** Hi Palak Mathur,
>
>> it seems overwhelming to have 10 directories. I am not saying it is not good just that I will not be able to handle those.
>
> I didn't anticipate this problem.  Maybe practicing with Treefactor
> and Dired would build this muscle over time.
>
> The rules are written to be straightforward at filing time.  One need
> only consider one object at a time.  Cascade filing means one need
> only compare the object with one directory at a time.  The first match
> wins.  I should emphasize that in the docs.
>
> Having all your headings jumbled into three huge files sounds like a
> state of permanent intractable overwhelm to me.
>
> ***** Hi Jean Louis,
>
> You are using Hyperscope as your primary PIM but integrating it with
> Org, and it sounds like you're using PostGreSQL and the directory tree
> together somehow.  I don't fully understand it.
>
> Clearly a database can do more than a directory tree alone.  The cost
> is that a database is more complex to use and maintain.  So that which
> can be handled by directory tree alone, should be.
>
>> I can find a mining engineer in country Senegal if I wish so, that has some work experience and I can see files pertaining to this person. But not that I would make file system directory Senegal to find the files for this person
>
> Of course not.  You would find a person under his name, not his
> country.  The person can move to a different country, after all.  At
> most you might link him to the country, as part of a list of people
> from X country.  But that is something better handled by a real
> database.
>
> To clarify, Treefactor is just an Emacs package with some minimal
> opinions.  10 Bins is an opinionated directory tree template.  Neither
> requires the other, but they're both part of Cyborganize, my overall
> PIM and publishing system.

Re: One vs many directories

[Jean Louis]

* Ihor Radchenko <yantar92@gmail.com> [2020-11-23 08:43]:
> >> I am wondering what you mean by Org's philosophy. Why would it have anything to do with directories?
> >
> > Org's philosophy is to have one or a handful of directories without
> > nesting of directories.  Users are not expected to have their Org
> > files in a deeply nested tree.  Org also prefers big files with large
> > trees rather than lots of little files.
> >
> > By philosophy, I mean the dev consensus on the correct way to do
> > things, and coded configuration and usability biases.
> 
> I believe that org support all possibilities. The user can decide to
> keep many (possibly nested) org files, a few large org files, or
> anywhere in between. There are several parallel feature sets allowing to
> work in a single file as well as with a bunch of smaller files.

Yes, sure, and I guess you mentioned some people have problems with
many files. And I have no problem at all with many files as they are
per subject separated, per person or per subject separated. They are
not hyperlinked to each other, it is me who make system to hyperlink
to files.

Searching for Joe Doe, F4 and I am in Org file for Joe Doe. My
personal TODO list need not really show the tasks assigned to Joe Doe,
I could show only * TODO Joe Doe and when I click there then I can get
all tasks for Joe Doe as new Org file.

It means I am accessing hundreds of Org files from the meta level by
using conceptual location backed by the database.

Some people maybe access multiple Org files through Agenda, me I
don't. Some items are "non existent" and I do not know how to ask
agenda to refresh itself. This is not big deal as I do not access
items throgh Agenda, though I find it very useful. 

org-agenda is trying to put all tasks and notes from various files
into one list and that is of course not so easy task considering that
files can be anywhere on the file system and that they need to be
"remembered". 

> For a single file, the user can search headings with org-goto (without a
> need to explicitly travel through all the nesting headline levels),
> reveal only headings satisfying certain keyword/tag/any other search
> criteria with org-sparse-tree, or built agenda views restricted to a
> single file (or even subtree).

M-x org-goto is useful feature to find headlines. And I never use it,
just standard Emacs search is enough within a file. Meanings I am
searching are often inside of the headline. And it is not my perosnal
way locating things.

Personally, I am using parts of Org, like specific headling to export
it and to send to remote person, or to print the file as project and
to bind it nicely.

When I see repetitive action, for example that I have to send "Daily
Report" template to a person by email, than I just bookmark that in
Emacs with {C-x r m} under something that I think is the meaning of
it. Then I forget about it. Next time when I need to send report, I am
{C-x r b} and quickly completing it and then I am exporting and
inserting into the email.

In general I speak of subsets or sub-lists among lists. List of Org
files is one list, list of headings within Org file is other list, and
list of specific subject related headings or bookmarks to such is
third subset of lists.

> For multiple files located anywhere in the filesystem, there is always
> org-refile capable of filing the information to proper place
> searching deeply nested headlines with ease regardless of the file the
> information is physically located in. Headlines from multiple files can
> be grouped using agenda views for any given search criteria (showing
> todo items or items for a single day/week is just a tiny subset of what
> agenda can do).

That may be useful for those who find it while my use case is
different, here is how it is for me:

** TODO Heading [1/2] [50%]

1) [X] Do this
2) [ ] Do that

that is my personal use case. I do not do things like:

** TODO Do this

Description of the task

And so far I know org-refile works on headings. It does not work on
list items.

Sometimes the task describes something that belongs to other file, I
just kill and yank to other file. And I keep RCS revision control
system of files.

As user may have many various sparse tasks to do or notes that require
action and attention in soonest future it is best to consolidate tasks
into one centralized system.

Such system should encompass all tasks or notes that require attention
or action in soonest future and should offer constant reminders to
user on what has to be done and when and which people are related to
the task.

When I mentioned "sparse tasks" I refer to my usage and handling of
mess:

1) Bunch of Org files, org-agenda and Org mode tries to accommodate me
   by consolidating everything into lists

2) There are hundreds of such tasks all over, Org tries to consolidate
   it.

3) There are various tasks and actions to do that are not recorded in
   Org files, those cannot be handled by Org.

4) There are database based Tasks in several groupings, some are just
   tagged with TODO, some are recorded in actual action requiring
   groups.

Instead of attempting to be perfect by using Org files for me
personally is best to consolidate all tasks in the database as such
are related either to people mostly or to some subjects related to me
personally which again belongs to "People".

And Org file for people need not have a task there, it can be exported
automatically into the Org.

- when searching for Person Robert S., I locate person and press F4

- Org file for that person is automatically created

- heading * Tasks is automatically created and expanded there by using SQL

Concept is here:

* Tasks

#+BEGIN_SRC sql :engine postgresql :exports results :results value raw 
SELECT '** [[(todo ' || simpletodos_id || ')][TODO]] ' || simpletodos_datecreated::date || ' ' ||
simpletodos_name || E'\n\n' || simpletodos_description FROM simpletodos
WHERE simpletodos_contacts = 23187;
#+END_SRC

#+RESULTS:
** [[(todo120)][TODO]] 2017-11-07 Do something

Something I need to do. (THIS IS HEADING OR TASK DESCRIPTION).

The SQL query need to defined only one time and not in the Org file,
but in the program that creates the Org file for the user. Instead of
the SQL query in the specific Org file it could be a simple Emacs
expression that never changes such as (tasks-for-user)

Then the SQL query generates the headings under #+RESULTS: and those
headings come from centralized consolidated database of tasks.

It is then interesting that this column of the database can include
any kind of the mode that Emacs supports, it could include any kind of
file, be it image, video or anything as long as such task is assigned
to the user. It could include other Org files and collection of Org
files or just description or database based whole Org file if
necessary. It becomes very abstract and liberated from limits.

That way I would be editing tasks on the meta level by clicking on
TODO link. If task would be done and completed it could be shown in
separate section of Org file related to user.

Additional buttons can be automatically included such as:

- All tasks for user -> Hyperlink to meta level consolidated TODO management

- Add task for user

- Re-assign from this user to other user

That is using Org mode as viewer of various information, not only as
handler of the information.

The cruft with org-agenda is then removed as then by pressing F5 I get
the list of all tasks and I could filter it how I wish. Which is
similar to org-agenda. The list is coming from the database and is
blazing fast.

Then I can filter using helm or ivy completion or built-in completion:

- I can simply type name of person related to the task, be it myself
  or somebody else. Majority of "my" tasks are related to other
  people. I search by people, sometimes by companies or organizations.

- I can type subject name or tag, any tag that need not be defined in
  single Org file and I find the task

- then if task is related to the person, I could quickly jump into
  persons meta report, from where there is Org files, other files,
  picture links all summarized in the meta Org profile, similar like
  FBI profiles we see on movies, one find the person's name and can
  see anything about that person.

- or I could delete the task, re-assign, modify the task. It becomes
  semi-automatically modified in the Org file of the user.

Can I automated the execution of Babel code upon opening of the Org
file?

When all tasks become centralized by any means, in my case in the
central database, then Org files get liberated from tasks and become
structural and relational. I can edit each task and I can always see
the updated list of the tasks and relations from one object to the
other. Hyperlinks get automatically created.

Personal problem is that tasks are sparse and separate in various Org
files and not centralized. I become dependent of org-agenda to do what
I need but it never does what I need.

Example is the need to relate task to people. Tasks are people related.

Purchasing cloths for your daughter? People.

Visiting museum? People related.

Some people are in Kenya, I do not think always on them. But I might
when I come there. Then I would write "Kenya" to get people I need to
put attention on. If tasks are centralized I can quickly jump to list
of Kenyan people.

If tasks are not centralized I need to put some tag under all those
people's files "Kenya" which is repetitive and error prone activity.

And I get trapped into "Org mode" for years. Which I am now, I am
trapped but it does not help me to manage things how I think.

I do value org-agenda but it is large effort to make SQL query out of
the Org mode. And very expensive effort as it is huge program:

-rw-r--r-- 1 415K Oct 19 10:20 org-agenda.el

If I organize all my tasks centrally and only display them in Org
files related to people, then making similar features like org-agenda
becomes trivial.

- Agenda for current week becomes trivial SQL query such as to select
  those tasks by current week. Result becomes a list that may be used
  either in Emacs completion or tabulated list mode or in Org or any
  kind of modes.

- To list entries with any kind of tags also become trivial, at least
  for centralized database tasks that have been tagged.

- searching for keywords is trivial single line SQL query.

- multi-occur is not trivial, as that would involve searching in all
  Org files. It asks for indexing Org files and going over them.

- Finding any FLAGGED entries would become trivial

It opens plethora of various means of reporting and accessing various
action related notes or tasks.

By general principle and due to nature of tasks that they do need
attention, I think that all tasks should be centralized, any how, it
does not matter how. From that principle I find it better that
majority of tasks are handled in just few files and not many files as
it becomes complex and users cannot any more remember which files.

org-agenda and Org try to remember that for user, but there are many
ways how Org files can be left without attention.

If my staff member in other country uses Emacs such can access
database and see assigned tasks. I can also export centralized tasks
for the user and send such to user as Org file.

If user updates tasks, as long as hyperlinks in the user's Org file
are not disturbed, I can review the work of the user and just click on
the hyperlink to update such task or mark them as DONE. Opinion of
staff member on what is done and what is not done is not necessarily
my opinion.

Description of content body of the heading can be programmatically
captured and entered into centralized database as a task.

Then we comes to actual execution of tasks. How do we get reminded?

Is the reminder only if I press {C-c a} for org-agenda? Do I need to
do action to get reminded?

I think computer shall be programmed to remind me on the start. Tasks
that are in the database can be shown directly in Emacs, but SQL
queries can be run and shown upon first log in into computer. Emails
could be sent, SMS could be sent to remind me or other people assigned
to the task. This brings attention and where people put attention that
is becoming reality.

Tasks could be shown by using timer in the mini buffer to remind user
of what has to be done and what is next, what is today to be done.

Keys can be assigned to TODAY, WEEK, MONTH, YEAR.

Birthdays can be consolidated automatically, as if there is database
of people, then their birtdays are automatic types of notes or tasks.

People who travel often would like to see other people in the city. By
changing one's location one could get list of known people and places
in the city. If I am in Mombasa, Kenya I may forget that good
restaurant as there are not many and visitor has to be brought in good
one, not bad one (which there are many).

With the introduction of emacs-libpq into GNU ELPA such features may
become reality.

emacs-libpq @ Github
https://github.com/anse1/emacs-libpq

Summary:

- tasks are people related

- plethora of Org files makes hard life to developers trying to satisfie users' needs

- tasks should be centralized and related to objects such as people, organizations, subjects

- Org files may be used as viewers for centralized task systems with
  all the features left as they are as all properties and tags and
  anything can be obtained from the database. Nothing changes.

- upon saving Org file or upon click or invocation, the task in the
  database can be updated if some description has been changed, but
  updates can take place in the database directly.

- tasks get liberated from any limits and formats, they can be just
  hyperlinks or hyperdocuments to just anything.

- integration and relation to many other objects becomes easier possible

- searching, indexing, making plans, or creating new meta Org files on
  the fly by using SQL queries becomes trivial. org-agenda of 450k can
  become easily redundant
  
- once centralized, there must be reminding system that works within
  or without Emacs, tasks are accessed by blazing speed, related
  people's files are quickly located, various queries and ways of
  reporting tasks in the list or meta Org file on the fly becomes
  possible.

I use "meta Org file" only to say that format is useful for displaying
structured and relational information for the reason that it has
hyperlinking support. As long as hyperlinks can be shown in any mode
then reports could be displayed any how. 

Re: One vs many directories

[Eric S Fraga]

On Tuesday, 24 Nov 2020 at 12:00, Jean Louis wrote:
> Can I automated the execution of Babel code upon opening of the Org
> file?

You can, by using file local variables.  For instance, for some files, I
do this:

#+begin_src org
  ,* local variables                         :noexport:
  # Local Variables:
  # eval: (org-sbe "startup")
  # End:
#+end_src

which will evaluate the named src block "startup" when file is opened.

Note that this is a potential security hole so only do this for files
you trust!

-- 
: Eric S Fraga via Emacs 28.0.50, Org release_9.4-118-g2a4578.dirty

Re: One vs many directories

[Jean Louis]

* Eric S Fraga <e.fraga@ucl.ac.uk> [2020-11-24 12:46]:
> On Tuesday, 24 Nov 2020 at 12:00, Jean Louis wrote:
> > Can I automated the execution of Babel code upon opening of the Org
> > file?
> 
> You can, by using file local variables.  For instance, for some files, I
> do this:
> 
> #+begin_src org
>   ,* local variables                         :noexport:
>   # Local Variables:
>   # eval: (org-sbe "startup")
>   # End:
> #+end_src
> 
> which will evaluate the named src block "startup" when file is opened.
> 
> Note that this is a potential security hole so only do this for files
> you trust!

For me is fine, as I do that for files I create.

When I have opened this email i was also asked to set local variables,
imagine. So that could maybe also mean that one could send email that
is constructed as Org file and if user answers YES, one could inject
malicious stuff.

--------------- the text above -------------------
still asks me if I like to allow eval: (org-sbe "startup")

So I think this is bug in Emacs as Local-variables should be on the
end of the file.

Re: One vs many directories

[Eric S Fraga]

On Tuesday, 24 Nov 2020 at 12:51, Jean Louis wrote:
> So I think this is bug in Emacs as Local-variables should be on the
> end of the file.

"end of file" is a rather loose term when it comes to local variables...
-- 
: Eric S Fraga via Emacs 28.0.50, Org release_9.4-118-g2a4578.dirty

Re: One vs many directories

[Diego Zamboni]

[-- Attachment #1: Type: text/plain, Size: 842 bytes --]

>
> So I think this is bug in Emacs as Local-variables should be on the
> end of the file.


According to the manual (
https://www.gnu.org/software/emacs/manual/html_node/emacs/Specifying-File-Variables.html#Specifying-File-Variables
):

The start of the local variables list should be no more than 3000
> characters from the end of the file


Given the length of the email, I guess this is why Emacs saw the variables
as being within the correct range.

--Diego


On Tue, Nov 24, 2020 at 12:57 PM Eric S Fraga <e.fraga@ucl.ac.uk> wrote:

> On Tuesday, 24 Nov 2020 at 12:51, Jean Louis wrote:
> > So I think this is bug in Emacs as Local-variables should be on the
> > end of the file.
>
> "end of file" is a rather loose term when it comes to local variables...
> --
> : Eric S Fraga via Emacs 28.0.50, Org release_9.4-118-g2a4578.dirty
>
>

[-- Attachment #2: Type: text/html, Size: 1634 bytes --]

Re: One vs many directories

[Jean Louis]

* Diego Zamboni <diego@zzamboni.org> [2020-11-24 16:13]:
> >
> > So I think this is bug in Emacs as Local-variables should be on the
> > end of the file.
> 
> 
> According to the manual (
> https://www.gnu.org/software/emacs/manual/html_node/emacs/Specifying-File-Variables.html#Specifying-File-Variables
> ):
> 
> The start of the local variables list should be no more than 3000
> > characters from the end of the file

I see and I wonder. I was thinking that not every prefix or comment
will be considered and that such must be really on the end of the file.

Re: One vs many directories

[Jean Louis]

* Diego Zamboni <diego@zzamboni.org> [2020-11-24 16:15]:
> >
> > So I think this is bug in Emacs as Local-variables should be on the
> > end of the file.
> 
> 
> According to the manual (
> https://www.gnu.org/software/emacs/manual/html_node/emacs/Specifying-File-Variables.html#Specifying-File-Variables
> ):
> 
> The start of the local variables list should be no more than 3000
> > characters from the end of the file
> 
> 
> Given the length of the email, I guess this is why Emacs saw the variables
> as being within the correct range.

Yes thank you. I was thinking Emacs will do that only in files where
it recognizes some comments or no comments and that variables need
to be pretty down in the file, on the bottom. Now I learn it is not
so.

That is security issue.

Re: One vs many directories

[Dr. Arne Babenhauserheide]

[-- Attachment #1: Type: text/plain, Size: 742 bytes --]


Jean Louis <bugs@gnu.support> writes:
>> The start of the local variables list should be no more than 3000
>> > characters from the end of the file
>> 
>> 
>> Given the length of the email, I guess this is why Emacs saw the variables
>> as being within the correct range.
>
> Yes thank you. I was thinking Emacs will do that only in files where
> it recognizes some comments or no comments and that variables need
> to be pretty down in the file, on the bottom. Now I learn it is not
> so.
>
> That is security issue.

Why is it a security issue? The variables do need to be close to the end
— 3000 characters is only about 50 lines.

Best wishes,
Arne
-- 
Unpolitisch sein
heißt politisch sein
ohne es zu merken

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 1125 bytes --]

Re: One vs many directories

[Jean Louis]

* Dr. Arne Babenhauserheide <arne_bab@web.de> [2020-11-24 21:51]:
> 
> Jean Louis <bugs@gnu.support> writes:
> >> The start of the local variables list should be no more than 3000
> >> > characters from the end of the file
> >> 
> >> 
> >> Given the length of the email, I guess this is why Emacs saw the variables
> >> as being within the correct range.
> >
> > Yes thank you. I was thinking Emacs will do that only in files where
> > it recognizes some comments or no comments and that variables need
> > to be pretty down in the file, on the bottom. Now I learn it is not
> > so.
> >
> > That is security issue.
> 
> Why is it a security issue? The variables do need to be close to the end
> — 3000 characters is only about 50 lines.

Emacs users, Org users on our mailing lists are not so private. Their
names and email addresses are in the public database. Spammer can
construct phishing type of an email, including something like Org news
or something and send such email to users. Among let us say 3000
people there will be percentage of users that will say Y to invoke the
local variables due to lack of knowing what is it doing to computer.

After that, anything becomes possible, including intrusion into
computer, capturing all email addresses, passwords, sending spam
emails from computer and so on.

Re: One vs many directories

[Tim Cross]

Jean Louis <bugs@gnu.support> writes:

> * Dr. Arne Babenhauserheide <arne_bab@web.de> [2020-11-24 21:51]:
>>
>> Jean Louis <bugs@gnu.support> writes:
>> >> The start of the local variables list should be no more than 3000
>> >> > characters from the end of the file
>> >>
>> >>
>> >> Given the length of the email, I guess this is why Emacs saw the variables
>> >> as being within the correct range.
>> >
>> > Yes thank you. I was thinking Emacs will do that only in files where
>> > it recognizes some comments or no comments and that variables need
>> > to be pretty down in the file, on the bottom. Now I learn it is not
>> > so.
>> >
>> > That is security issue.
>>
>> Why is it a security issue? The variables do need to be close to the end
>> — 3000 characters is only about 50 lines.
>
> Emacs users, Org users on our mailing lists are not so private. Their
> names and email addresses are in the public database. Spammer can
> construct phishing type of an email, including something like Org news
> or something and send such email to users. Among let us say 3000
> people there will be percentage of users that will say Y to invoke the
> local variables due to lack of knowing what is it doing to computer.
>
> After that, anything becomes possible, including intrusion into
> computer, capturing all email addresses, passwords, sending spam
> emails from computer and so on.

this is just baseless fear mongering based on nothing but speculation.

Of your suggested 3000 users, only a very small percentage use Emacs as
their mail client. Of those, only an even smaller number will have their
mail client configured to render messages with a mode that supports
local variables and even a smaller number of those would say yes to
executing the code. In fact, anyone who has gone to the extent of
configuring their Emacs email client to open messages with a mime type
of x-org (or even just based on an attachment with *.org in the file
name) is more than likely to be sufficiently technically aware not to
say yes when asked.

Few, if any user, is going to download some random attachment in an
email message, open it in emacs and then say yes to a message warning
them that doing so might be dangerous. Such ill-informed users have been
pretty much weeded out by all the other scam phishing out there by now.
To convince them to go through such steps would require some pretty
convincing content - a simple org news attachment or similar is unlikely
to do it.

Even if you do get them to say yes, they are still a long way from being
able to compromise the computer Emacs is running on. Gaining some level
of access is one thing, actually being able to do something with that
access is another. Trying to compromise a computer, which these days
typically involves privilege escalation, would be extremely difficult to
achieve with elisp. The best you could hope for would be to install a
trojan or back door what would allow the attacker to install other
tools. Could be possible, but is definitely non-trivial.

This ability has been around in Emacs for a very long time - more than
30 years, possibly even longer. There has not been a single recorded
incident of large number of users being compromised as a result of this
feature. I've not even heard of small numbers being compromised as a
result of this feature.

I'm sure you will disagree. My suggestion is that if you believe this is
a security issue, you put together a proof of concept to demonstrate the
vulnerability - this is how such security issues get resolved.
Demonstrate how the security issue can be exploited with actual proof of
concept code rather than mere speculation and that will provide
something concrete which can be dealt with. I suspect you will find it
much harder to achieve once you actually try to make it work.

--
Tim Cross

Local variables insecurities - Re: One vs many directories

[Jean Louis]

* Tim Cross <theophilusx@gmail.com> [2020-11-25 09:41]:
> >> Why is it a security issue? The variables do need to be close to the end
> >> — 3000 characters is only about 50 lines.
> >
> > Emacs users, Org users on our mailing lists are not so private. Their
> > names and email addresses are in the public database. Spammer can
> > construct phishing type of an email, including something like Org news
> > or something and send such email to users. Among let us say 3000
> > people there will be percentage of users that will say Y to invoke the
> > local variables due to lack of knowing what is it doing to computer.
> >
> > After that, anything becomes possible, including intrusion into
> > computer, capturing all email addresses, passwords, sending spam
> > emails from computer and so on.
> 
> this is just baseless fear mongering based on nothing but
> speculation.

My experience with such people come from last more than 25
years. CVE list is the reference I already quotes. Some thing were
improved like this:

https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=684695

So one should know how dangerous it was to introduce local variables
with possibility to automatically eval anything there.

> Of your suggested 3000 users, only a very small percentage use Emacs as
> their mail client.

Those which I know through Emacs list mostly use it. I am using it. Is
there any way to know who use and who not?

In general I am reading emails with Mutt, but I am answering with
Emacs. Sending emails is often by Emacs or by Mutt. I use sometimes
M-x rmail as well

> Of those, only an even smaller number will have their mail client
> configured to render messages with a mode that supports local

I have not configured anything. In fact I have opened the email and I
was surprised that I am getting those dialogues to execute local
variables. And I an in mail-mode now. Mutt opens new emacsclient and I
edit the file.

Obviously user does not need to configure anything. You maybe refer to
specific mode where it does not execute.

It will try to execute even if I open .TXT file.

The very design of Local variables I do not find trustful for these
explained reasons. I do not protest against it as now is too late, but
as I mentioned more educational references could be provided in the
dialogue that asks user to execute local variables or not.

> variables and even a smaller number of those would say yes to
> executing the code. In fact, anyone who has gone to the extent of
> configuring their Emacs email client to open messages with a mime
> type of x-org (or even just based on an attachment with *.org in the
> file name) is more than likely to be sufficiently technically aware
> not to say yes when asked.

I do not need to configure emacs to anything to get the local variable
execution dialogue, verify it on your side.

I can basically get any attachments by email, try to view them in
Emacs and it will execute.

> Few, if any user, is going to download some random attachment in an
> email message

Sorry, but I have no choice but to download all attachments. Majority
of email clients do not offer choice to download specific attachments
they download whole message as one.

> open it in emacs and then say yes to a message warning them that
> doing so might be dangerous.

It does not warn to be dangerous. There is no word of danger in the
dialogue.

I would rather like the dialogue to does what you say but it does not,
I would prefer it like this:

                 ===========
		  DANGEROUS
		 ===========

But it is not like that, and I have elaborated why it is not like
that. Text writers are not programmers. You assume that every user
starts from your viewpoint of 30 years experienced Emacs wizard. And I
am speaking of design view point. Emacs is still called "editor"
today.

The description of Emacs I get in Hyperbola GNU/Linux-libre is:

The extensible, customizable, self-documenting real-time display
editor, without D-Bus support

Nowhere it says in the description that it can potentially expose me
to malicious evaluations of software just by opening a text file. That
you know what Emacs is really and me too, it does not make it more
secure.

We make assumptions that we will know that users will be safe, but
that is wrong assumption and there would be no CVE as I have already
referenced it it it would be so.

> Such ill-informed users have been pretty much weeded out by all the
> other scam phishing out there by now.

I would not say so. As non-native English speaker to say for users
that they are ill-informed sounds to me not appropriate. We invite
users to use Emacs, they will download, open, and may be offered to
read the ebook or other interesting text, and text will ask them to
evaluate the variables. You say that the subset of those users who
will know what is "variable", "value", "evaluation", "safe" is small
and not important, and I think this is most important for the future
of next 100 years for Emacs being useful to many people. For that
reason I cannot call such users ill-informed, rather not informed and
coerced into decisions which they are not able to take as they are not
informed.

> To convince them to go through such steps would require some pretty
> convincing content - a simple org news attachment or similar is
> unlikely to do it.

Maybe you get informed, that is very easy, easier than you
think. There are many ways to do that and that is very simple by
tricking users, as Emacs related mailing lists are harvested and from
time to time users can be sent emails and emails can offer them
themes, programs, to trick them. It is very easy to convince users.

Examples from other computing areas on how users are tricked:
https://news.softpedia.com/news/Facebook-Users-Tricked-into-Loading-Malicious-Code-in-Their-Browsers-146604.shtml

https://techcrunch.com/2019/08/16/android-users-tricked-adware-apps/

https://www.independent.co.uk/life-style/gadgets-and-tech/news/google-chrome-adblocker-fake-download-scam-users-tricked-adblock-plus-extension-a7992711.html

https://www.dailymail.co.uk/sciencetech/article-4682528/Facebook-users-tricked-sharing-hoax-message.html

https://www.offthegridnews.com/privacy/facebook-users-give-personal-data/

> Even if you do get them to say yes, they are still a long way from
> being able to compromise the computer Emacs is running on.

I cannot be sure to what you refer. If email is sent to 10,000 people
there will be subset of people who will say yes, and what is
programmed in the local variables could be anything.

> Gaining some level of access is one thing, actually being able to do
> something with that access is another. Trying to compromise a
> computer, which these days typically involves privilege escalation,
> would be extremely difficult to achieve with elisp.

Fetch from Internet and execute. That is what majority of intruders
do. If you are not intruder you cannot know. There are rootkits
everywhere on Internet. Fetch and execute. Sometimes simpler than that.

> The best you could hope for would be to install a trojan or back
> door what would allow the attacker to install other tools. Could be
> possible, but is definitely non-trivial.

Exactly. Experience over 18 years with server administration tells me
about that. 

> This ability has been around in Emacs for a very long time - more than
> 30 years, possibly even longer. There has not been a single recorded
> incident of large number of users being compromised as a result of this
> feature.

Also the intrustions that have taken place on hosting servers I have
not reported anywhere. It is not relevant really and I have explained
that when intrusion does happen user may not go to report to the list
or connect to experienced hackers. On the other hand user is faced
with a dialogue that requires user to know what is programming,
variable, value and safe. Users do not know.

Today I was watching Ralph that broke the Internet and somebody
mentioned to him "safe" and he said he feels safe. That reminds me of
that dialogue. Something contains values that may not be safe? Well I
am safe, damn YES.

When expressing "safe" one has to say about the context, safe from
what? 

> I've not even heard of small numbers being compromised as a result
> of this feature.

You can as well make program popular and root and need not hear of it
ever. There is plethora of programs that one does not hear of it, why
should you hear of it? When there is really good way to intrude into
people's computers those are not published for everybody to
know. Facebook data was downloaded continually by intruders back in
time. I remember before 16 years a simple website that I reference to
a friend in the same room could give me access to all his files. It
was amuzing. Today there are many problems with Javascript and Emacs
is just maybe not so wide spread to invoke public incidents. The CVE
list should be enough to say that there are security incidents coming
again and again.

> I'm sure you will disagree. My suggestion is that if you believe
> this is a security issue, you put together a proof of concept to
> demonstrate the vulnerability - this is how such security issues get
> resolved.

Just put eval: do anything malicious. That is simplest. Fetch URL,
save, chmod, execute. Anything can be there. 

> Demonstrate how the security issue can be exploited with actual
> proof of concept code rather than mere speculation and that will
> provide something concrete which can be dealt with. I suspect you
> will find it much harder to achieve once you actually try to make it
> work.

Well, you say so, but this one worked pretty well:

# Local Variables:
# eval: (progn (delete-file "/tmp/new/new") (message "Done"))
# End:

I could as well send files to myself by using their email system
whatever is there like mail, mailx, mutt, maybe Emacs stuff as
well. Or I could send me their password store, I could get
~/.authinfo, or maybe even test if sudo works without password and
then place my insecure DNS servers and redirect to phishing
websites. Just anything becomes possible with Emacs variables.

Re: Local variables insecurities - Re: One vs many directories

[Eric S Fraga]

On Wednesday, 25 Nov 2020 at 15:38, Jean Louis wrote:
> I have not configured anything. In fact I have opened the email and I
> was surprised that I am getting those dialogues to execute local
> variables. 

Very strange.  It was my email that instigated this part of the
thread.  I can view my email and I do not get asked to set any locate
variables or, in this case, evaluate the elisp expression.  Out of
curiosity, what mail reader did you use that actually interprets file
local variables?  Gnus article view does not.

-- 
: Eric S Fraga via Emacs 28.0.50, Org release_9.4-118-g2a4578.dirty

Re: Local variables insecurities - Re: One vs many directories

[Jean Louis]

* Eric S Fraga <e.fraga@ucl.ac.uk> [2020-11-25 16:06]:
> On Wednesday, 25 Nov 2020 at 15:38, Jean Louis wrote:
> > I have not configured anything. In fact I have opened the email and I
> > was surprised that I am getting those dialogues to execute local
> > variables. 
> 
> Very strange.  It was my email that instigated this part of the
> thread.  I can view my email and I do not get asked to set any locate
> variables or, in this case, evaluate the elisp expression.  Out of
> curiosity, what mail reader did you use that actually interprets file
> local variables?  Gnus article view does not.

I use Mutt.

The message is opened in Emacs in mail-mode

Then I have been testing and even text files invoke local variables.

When I send messages I often send it from Emacs, but that is different
subject.

Re: Local variables insecurities - Re: One vs many directories

[Eric S Fraga]

On Wednesday, 25 Nov 2020 at 16:13, Jean Louis wrote:
> I use Mutt.
> The message is opened in Emacs in mail-mode

Ah, so mutt saves content in a file which is then opened by
Emacs.  Okay, that makes sense.  Gnus does things the other way around:
opens the buffer (associated with a file in the draft directory),
inserts the content, and then puts the user in control.  File local
variables don't get a chance to be interpreted then.

> Then I have been testing and even text files invoke local variables.

Yes, of course.  That's the whole point?

(and, yes, I've been reading the thread so I know the concerns about
security etc.)

-- 
: Eric S Fraga via Emacs 28.0.50, Org release_9.4-118-g2a4578.dirty

Re: Local variables insecurities - Re: One vs many directories

[Jean Louis]

* Eric S Fraga <e.fraga@ucl.ac.uk> [2020-11-25 16:58]:
> On Wednesday, 25 Nov 2020 at 16:13, Jean Louis wrote:
> > I use Mutt.
> > The message is opened in Emacs in mail-mode
> 
> Ah, so mutt saves content in a file which is then opened by
> Emacs.  Okay, that makes sense.  Gnus does things the other way around:
> opens the buffer (associated with a file in the draft directory),
> inserts the content, and then puts the user in control.  File local
> variables don't get a chance to be interpreted then.

That is specific to Gnus. Any file opened by Emacs any will still
invoke the dialogue for local variables.

> > Then I have been testing and even text files invoke local variables.
> 
> Yes, of course.  That's the whole point?

You know that point is bad design and assumption that every user is
programmer who knows what are local variables and what are definitions
of Emacs functions, it is incredible situation.

Re: Local variables insecurities - Re: One vs many directories

[Tim Cross]

Jean Louis <bugs@gnu.support> writes:

> * Eric S Fraga <e.fraga@ucl.ac.uk> [2020-11-25 16:58]:
>> On Wednesday, 25 Nov 2020 at 16:13, Jean Louis wrote:
>> > I use Mutt.
>> > The message is opened in Emacs in mail-mode
>>
>> Ah, so mutt saves content in a file which is then opened by
>> Emacs.  Okay, that makes sense.  Gnus does things the other way around:
>> opens the buffer (associated with a file in the draft directory),
>> inserts the content, and then puts the user in control.  File local
>> variables don't get a chance to be interpreted then.
>
> That is specific to Gnus. Any file opened by Emacs any will still
> invoke the dialogue for local variables.
>
>> > Then I have been testing and even text files invoke local variables.
>>
>> Yes, of course.  That's the whole point?
>
> You know that point is bad design and assumption that every user is
> programmer who knows what are local variables and what are definitions
> of Emacs functions, it is incredible situation.

I guess this is probably the main point where we disagree.

Emacs is first and foremost a programmers editor. It was never designed
as a general purpose editor, but rather specifically as an editor for
programmers.

If you jump into a formula 1 race car, you would find it almost
impossible to drive. The gearbox would be unfamiliar and difficult to
use, the clutch would be difficult to use etc. If you got it going, you
would have a high likelihood of crashing. Luckily, you would probably
just stall and get nowhere.

Is this the fault of the design of the race car or the driver? Would it
make sense to change the design of a race car to use a standard gearbox
and clutch just because at some point someone who is not a race car
driver might want to drive it?

Are there risks associated with local variables. Yes. Is there
sufficient protection against these variables being used for malicious
purposes in Emacs. I think the answer is yes. Is there any evidence of
these variables being used for malicious purpose. None that I am aware
of. Has there been bugs in the implementation of this facility - yes.
Have these bugs been addressed once identified - yes.

With respect to your email example, the number of people who are exposed
is even less - it is really only those who are using it in the same
manner as you. That is, where they have configured their mail client
(such as Mutt) to use Emacs as the external editor. None of the Emacs
mail clients I have used do this (this includes VM, mu4e, gnus,
wonderlust and mew).

anyone who has gone to the effort to configure their mail system to use
an external editor and who then answers yes to the statement
"...contains values that may not be safe. Do you want to apply it?" is
someone with inherently unsafe practices. I doubt any change in wording
or phrasing would be of any help for them. However, the correct way to
deal with this would be to offer up a patch to the Emacs developers
which improve this wording. I would suggest the set of people who are
technically aware enough or have sufficient technical interest to have
adopted emacs as their email viewer and who would still answer yes to
any dialogue warning them of unsafe actions when opening content from an
unknown source is very small.

Local variables is a powerful and useful feature. Like many powerful
features, it can be abused. We differ in our opinions on whether those
safe guards are sufficient. I believe they are and I believe you are
over stating the risks. I don't believe we will arrive at any consensus
and I feel this thread has run its course. You are of course free to
respond, but I will refrain from further participation as this has
wondered off topic for org mode and I see little to be gained from
further back and forth.


--
Tim Cross

Re: Local variables insecurities - Re: One vs many directories

[Jean Louis]

* Tim Cross <theophilusx@gmail.com> [2020-11-25 23:54]:
> I guess this is probably the main point where we disagree.
> 
> Emacs is first and foremost a programmers editor. It was never designed
> as a general purpose editor, but rather specifically as an editor for
> programmers.

Yes. And when I was born as baby I was designed for milk, not for
typing, times change. People use GNU/Linux and Emacs is not advertised
as programmers or exclusively programmers editor. Some other editors
are advertised that way. So think how many hundreds of thousands of
users are working with Emacs.

Here is how Debian GNU/Linux describes it:
https://packages.debian.org/buster/emacs

If there are 10 programmers there are probably 100 if not 500
non-programmers.

> If you jump into a formula 1 race car, you would find it almost
> impossible to drive. The gearbox would be unfamiliar and difficult to
> use, the clutch would be difficult to use etc. If you got it going, you
> would have a high likelihood of crashing. Luckily, you would probably
> just stall and get nowhere.
> 
> Is this the fault of the design of the race car or the driver?

Race cars are not distributed through GNU/Linux operating systems and
are not easily downloadable by everybody, in general, they are also
expensive. While it all sounds entertaining, Emacs is not a race
car. And we cannot say to users not to use it if they are not Formula
One Drivers.

> With respect to your email example, the number of people who are exposed
> is even less - it is really only those who are using it in the same
> manner as you. That is, where they have configured their mail client
> (such as Mutt) to use Emacs as the external editor. None of the Emacs
> mail clients I have used do this (this includes VM, mu4e, gnus,
> wonderlust and mew).

I do not need to use Emacs with Mutt to invoke local variables. I can
get files by any means and by any opening of the file with Emacs it
will be invoked. Somebody could send me file to download and
open. File can come from anywhere, it is not Mutt related really.

Gnus buffers and email clients do not invoke local variables and that
is fine. But security issue is not email centric, but file centric.

> anyone who has gone to the effort to configure their mail system to use
> an external editor and who then answers yes to the statement
> "...contains values that may not be safe. Do you want to apply it?" is
> someone with inherently unsafe practices.

That is very rigid assumption. People set editors for various email
clients since decades. Try to think from another people's view points.

Here is example:
https://stackoverflow.com/questions/15865495/opening-a-file-in-emacs-values-that-are-not-safe

That person has quite different view point. Person asks "Why it would
not be safe?" and one should know when one person writes there for an
answer there are probably thousand other persons who did not write for
the answer.

Other person asked:

"Thanks, that's very helpful. Why would a file (i.e. the author of the
file) require or ask Emacs to apply configuration values when just
opening/visiting the file? – Amelio Vazquez-Reina"

I know why, but people using Emacs are asking why. Many will not ask
and will say, damn YES, as I feel safe!

Denial of Service Attacks possible:
https://github.com/aquamacs-emacs/aquamacs-emacs/issues/147
https://gitmemory.com/issue/davidswelt/aquamacs-emacs/147/478196367

.emacs considered not safe:
https://www.cs.ait.ac.th/~on/O/oreilly/tcpip/puis/ch11_05.htm

OK then now back to Org discussions.

Jean

Re: Local variables insecurities - Re: One vs many directories

[Tom Gillespie]

Hi Jean,

Some points in summary before a long email.
1. Having an accepting default behavior as a user (i.e., saying yes to
   all prompts) is bad security practice. The only thing that systems
   can do is prompt as infrequently as possible in hopes that people
   don't get prompt fatigue. Emacs does this.
2. If mutt is launching Emacs, you can pass --eval "(setq
   enable-local-eval nil)" on the command line and all file local
   variables will be ignored and treated as plain text.
3. If people don't read the manual and don't read the prompt, there
   isn't much more that Emacs can do while still empowering the
   user. In those cases we also can't assume that the community will
   be of much help either, as we are trying to be here.
4. That said, the local variables prompt could be more alarming and
   provide a quick way to access the manual about local
   variables. I'll look into a patch for that.

Now on to the rest!

Full disclosure. I make use of file local variables every day
and I have spent quite a while writing orgstrap which relies heavily
on them, so I am definitely biased.

Emacs is a sharp tool. Experts need sharp tools. If someone complains
about the security of prompts but also admits that they always say yes
to prompts without reading them, then no one will take them seriously
when they try to argue that it is the prompt that is insecure. It is
like complaining that the forest is dangerous while insisting on
eating the berries of all the plants and picking up all the snakes.
The forest is dangerous, but not merely because the berries and the
snakes exist in it. Heck, some snakes even try to warn you! There are
some rattlesnakes that have evolved to not rattle because idiot humans
go find and kill them. Now everyone suffers because there are silent
rattlesnakes that will strike without warning.

Degrading usability because some users are irresponsible just
reinforces the irresponsible behavior and everyone loses. The endgame
for all prompts is a two-man two-key system where the switches are
separated by 30 feet and must be turned within 1 second of each
other. How else could we be sure that the user really meant to say
yes!? Emacs is scary, but not nuclear launch system levels of scary.

Furthermore, Emacs does try to account for users who don't read the
manual and has sane and safe defaults. Namely it does not blindly
execute code but prompts the user and lets them know that something is
going on. Further, when it prompts the user it does not provide a
default action, it forces them to answer so that they must attend to
what they are doing. This provides the user with an opportunity to
become aware of a feature of Emacs that is new to them. The only other
option is to never execute local variables until a user reads the
manual and changes their config. Such a design infantilizes the user
and does not empower them.

Emacs isn't just a dissociated tool, or at least it tries not to be
(lots of people also complain about the splash screen being enabled by
default!). The Emacs community often also goes out of its way to share
its knowledge when such situations arise (as we are doing here). We
can't reach everyone (yet!) but many people donate time and effort to
share. Emacs and Org both actively encourage users to participate in
the mailing lists because venues like StackOverflow are not guaranteed
to be seen by the community and are not officially supported.

Finally on this point, if we are willing to open the manual we see
that Emacs tries to educate users about this, the first line of the
section on file local variables is "File-local variables can be
dangerous."

With regard to the local variables prompt. Maybe the right thing to do
in this case would be to add a "?" option so that users can open the
manual? That seems like it would help. That said, the second line of
the prompt reads "contains variables that may not be safe (*)" which
should be alarming. I guess it could be changed to "THIS FILE COULD
PWN YOU. Please review potentially unsafe variables marked with an
asterisk (*)." Or just make the second line a bold warning? I have a
patch I need to send in to make it possible to use lexical scope in
eval local variables, I'll take a look at this while I'm there.

It is forgivable to not be utterly terrified of systems that can
execute arbitrary code, given that they mostly look like rocks, and we
are surrounded by them all day. However, they are utterly terrifying
existences! While I think it would be great for the Emacs splash
screen to come with a big warning that says "WARNING THIS PROGRAM
ENABLES YOU TO EXECUTE ARBITRARY CODE" I admit that it took me nearly
a decade to begin to understand what arbitrary code execution means
and why I should be scared of it. Those words are meaningless to a 12
year old who at best will think "What is the big deal about arbitrary
code? I can already run any code that I want!"

This is why I pointed you to orgstrap. The eval: (org-sbe startup)
local variable is utterly terrifying to me. I added a layer that
allows me to know that I have audited the block that I'm about to
execute by calculating the checksum of the elisp. Now we have an
audited code evaluation system. Much less terrifying, with a chance to
prevent nasal demons rather than playing prompt fatigue roulette and
hoping it is never my turn. I can't read the instructions for you nor
explain better than the link itself https://github.com/tgbugs/orgstrap
why you might want to use it if you care about security while still
wanting to run code to simplify complex processes.

You ask why people need local variables in the first place. The answer
is partially because they got sick of getting emails from people who
had misconfigured systems and didn't read the manual and the
instructions for how to properly configure them to run a file (sound
familiar?). They also got sick of people screwing up the indentation
of files and not following coding conventions, so they got the
computer to do it for them because it is way more effective than
trying to do it with politics.

People who don't follow instructions don't follow instructions. This
seems like an obvious statement, but its consequences are nearly
impossible to deal with and there is not much we can do about it other
than to reduce the number of instructions we have to give. Local
variables are a great way to reduce the number of instructions. I have
tested this (unscientifically) and I can report that error rates are
lower and users are happier when they can accept local variables and
let the author of the file worry about the details. Hard to follow
instructions waste everyone's time if they are followed at all. There
are better workflows, but they are not foolproof and not obvious. How
could they be? Few to none are born knowing cryptography much less the
best practices for how to employ it.

People put up fences and warning signs, and some folks will still
climb over them. I suppose we could go back to only using Turing
incomplete systems and languages.  Those are probably safe.

As alluded to in one of my previous emails, questioning the existence
of a feature is often a sign to slow down and take the time to
understand what you might be missing. Sometimes there is nothing
there, but in Emacs that is rarely the case. To quote myself "Emacs is
useful because countless others have encountered problems the likes of
which a single person only rarely dreams. Not only that but they have
probably already implemented a solution that is fairly easy to find
once you encounter the problem and in 80% of the cases can be used
unmodified."  There are features of Emacs that I suspect I will never
know about no matter how long I use it and that is as it should be.

There are members of the larger community who I suspect have seen
countless users, some new, some veteran, come to question and complain
about a feature that they rarely personally use and thus do not
understand. Such observers know from experience that there are likely
thousands of other users for whom that very feature is a critical part
of their daily workflow. Someone posted to this list a few months ago
suggesting that org be stripped of all the complexity and broken into
smaller projects because they didn't see the need to include things
like org-babel in the core.  Fortunately the community has long
experience with these kinds of limited perspectives and knows how to
deal with them, but they cannot simply be ignored.

Best,
Tom

Re: Local variables insecurities - Re: One vs many directories

[Jean Louis]

* Tom Gillespie <tgbugs@gmail.com> [2020-11-26 05:07]:
> Hi Jean,
> 
> Some points in summary before a long email.
> 1. Having an accepting default behavior as a user (i.e., saying yes to
>    all prompts) is bad security practice. The only thing that systems
>    can do is prompt as infrequently as possible in hopes that people
>    don't get prompt fatigue. Emacs does this.

I know, and do not speak for one person but for those who will not
know. To ask users who do not know programming using editor for text
editing and to assume that users will know programming terms:
variables, values, applying, safe and ANYTHING else that is written in
eval: is bad security practice.

> 2. If mutt is launching Emacs, you can pass --eval "(setq
>    enable-local-eval nil)" on the command line and all file local
>    variables will be ignored and treated as plain text.

Sure, I know, but human text editors will not know. They are faced
with YES/NO. What has to be improved is the YES/NO dialogue that has
no hyperlinks to its corresponding explanations and asks users things
with wrong assumption that user will understand them.

> 3. If people don't read the manual and don't read the prompt, there
>    isn't much more that Emacs can do while still empowering the
>    user.

Empowering can take place in the dialogue as such has enhancement
space. Do you think it is good idea?

>   In those cases we also can't assume that the community will be of
>   much help either, as we are trying to be here.

Community is of great help. Users are many and just fraction of users
are in this community for example. Only subset of users even while
being in community or reading emails or website will be participating
in such.

> 4. That said, the local variables prompt could be more alarming and
>    provide a quick way to access the manual about local
>    variables. I'll look into a patch for that.

Yes, that is it. To empower user is to give him straight better
reference on what to do.

Following places could become buttons:

The local variables list in new-concept.org
    ^^^^^^^^^^^^^^^
    Button to A below
    
contains values that may not be safe (*).
^^^^^^^^^^^^^^^                 ^^^^
Button to A below               Button to A below

Template:

Do you want to apply it?  You can type
               
y  -- to apply the local variables list.
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
	 Button to A below
	 
n  -- to ignore the local variables list.

!  -- to apply the local variables list, and permanently mark these
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
	 Button to A
      values (*) as safe (in the future, they will be set automatically.)

  * eval : (let ((pos (org-babel-find-named-block "stages"))) (when
    pos (save-excursion (org-with-point-at pos
    (org-babel-execute-src-block)))))
    ^^^^^^
    Button to A below.

The template how it looks originally:

The local variables list in new-concept.org
contains values that may not be safe (*).

Do you want to apply it?  You can type
y  -- to apply the local variables list.
n  -- to ignore the local variables list.
!  -- to apply the local variables list, and permanently mark these
      values (*) as safe (in the future, they will be set automatically.)

  * eval : (let ((pos (org-babel-find-named-block "stages"))) (when pos (save-excursion (org-with-point-at pos (org-babel-execute-src-block)))))

Button A should follow here:
^^^^^^^^^
File: emacs.info,  Node: Safe File Variables,  Prev: Specifying File Variables,  Up: File Variables

49.2.4.2 Safety of File Variables
.................................


Right now situation is this:

- if user press C-g to abort the file will be loaded, which is good

- user will not be able to access any of the visible buttons by using
  cursor because somebody made cursor invisible in that
  dialogue. Cursor should be made visible that keys can be used to
  access buttons

- There shall be on minibuffer some key like "? TO READ MANUAL" to go
  straight to the manual section above which is warning user much
  better.

- IF THE USER press C-g, or tries to escape it or clicks by using
  keyboard on the button, or even switch to the buffer with keyboard,
  or uses mouse to activate the button, or uses ? to activate to read
  manual, in all those cases the dialogue should be interrupted and
  file loaded. User should not be left facing dialogue if it was clear
  from user's interaction that there was doubt with the meanings of
  the dialoue.

- Adding references on unsafe local variables to TUTORIAL

> Full disclosure. I make use of file local variables every day
> and I have spent quite a while writing orgstrap which relies heavily
> on them, so I am definitely biased.

With such enhancement there is no need to change history of the past,
we just change the future and history for the future.

> Emacs is a sharp tool. Experts need sharp tools. If someone
> complains about the security of prompts but also admits that they
> always say yes to prompts without reading them, then no one will
> take them seriously when they try to argue that it is the prompt
> that is insecure.

I never say YES by the rule. I have explained sincerely that I did say
YES for number of years as I was thinking those are enhancements that
make things work. And I have used Emacs in that way for long
time. Overall I did not encounter many files with local variables
because I did not read other people's files for long time. Please try
to get the perspective. I was programming myself but in other
languages and never used local variables. If I do not read other
people's files it is hard to encounter Local variables. When it came
about 2016 to start reading other people's files that is where I
rather pressed YES. I did not put attention on each function and those
meanings and did not know meanings of functions.

If you say that it is important to take users seriously, then take
their behavior with the editor seriously and their proneness to
failures as well. Do not expect power users.

> It is like complaining that the forest is dangerous while insisting
> on eating the berries of all the plants and picking up all the
> snakes.

I think that analogy of Tom with the racing car is better.

To make your analogy better it would be like you have the forest with
excellent mushrooms and you invite general public to the forest. Then
you face public with some of potentially poisonous mushrooms and you
start asking them if they wish to accept Amanita muscaria mushroom as
it could be possibly unsafe. But people came to see those potentially
good mushrooms and did not hear of posionous mushrooms, damn, let me
say YES, I am here and I feel safe!

> Degrading usability because some users are irresponsible just
> reinforces the irresponsible behavior and everyone loses.

Usability is already degraded. You can enhance it. It is not nice
calling users irresponsible just because they did not read the manual
and fully understand it. In addition to Emacs Manual by asking users
to decide if variables are safe or not, Emacs is in the same time
asking users to know meanings of ALL possible functions and variables
that are located in such files. That is one big chunk of assumptions
and conditions that user should become responsible by such viewpoint.

In that case it would be best to lock the Emacs of using it unless ALL
users can pass the test to know everything written in the Emacs manual
and Emacs Lisp manual as well.

> Furthermore, Emacs does try to account for users who don't read the
> manual and has sane and safe defaults. Namely it does not blindly
> execute code but prompts the user and lets them know that something is
> going on. Further, when it prompts the user it does not provide a
> default action, it forces them to answer so that they must attend to
> what they are doing. This provides the user with an opportunity to
> become aware of a feature of Emacs that is new to them.

While that purpose is good and that is same purpose that I am
supporting our opinions differ.

Do you see how you said: "This provides the user with an opportunity
to become aware of a feature of Emacs that is new to them." -- this is
what is my wish too. I just think that one cannot become aware of a
feature that is new to them.

> The only other option is to never execute local variables until a
> user reads the manual and changes their config.

That is optional only in historical terms or if we make time machine
to go back and make more future prone defaults.

> With regard to the local variables prompt. Maybe the right thing to do
> in this case would be to add a "?" option so that users can open the
> manual?

Button and ?

? was or is at most cases shown to user, but on the dialogue speaking
of safety is not shown. 

> That seems like it would help. That said, the second line of
> the prompt reads "contains variables that may not be safe (*)" which
> should be alarming. I guess it could be changed to "THIS FILE COULD
> PWN YOU. Please review potentially unsafe variables marked with an
> asterisk (*)."

You could include that for advanced users. The point of discussion is
that millions are not advanced users and using terminology such as
variable, value, apply, safe is out of the context for the user who is
not advanced. Using "pwn" would be hard to understand.

When word is out of the context such cannot be understood.

I gave one reference from Stack-something where users do not
understand "safe" and then other one came with the comment also not
understanding "safe".

Safe in the context of the dialogue and relation to computing means
safe to user's data, privacy and safe computing. One has to know
that majority of computer users are not aware of security problems in
computing and that they put things on risk.

Users who are not programmers or just know how to edit files,
translate, make reports, they need not know what is programming and
computer terminology. If that would be so than there would be no
computing dictionaries, the manual and Emacs Lisp manual just to name
few examples.

I have my staff members who are using Emacs. They do not know it. One
is in Tanzania, one is in Kampala, Uganda. They will know how to edit
Org file and how to send me portion of it, but they did not learn
nothing about computing.

For them the word "safe" is never in the context of computing! Because
if one does not know the context around the word meaning is lost. For
them "safe" means free from danger or the risk of harm. That is in the
personal context, did anybody tell them there is something in the
computing context? References from Stack- show example that people
understand it so. When saying "safe" one has to say "safe to your
data" and if there are no references to context then there shall be
references to manual as example.

I am not sure if "bold" is right way to warn users if it does not work
well on console. Underlined, different colored, links are showing
better in console and if somebody does not use colors I think
underlined will show better.

So if you can do something about that to enhance references to
meanings of what dialogue is telling to user, that would be great.

Jean

Re: Local variables insecurities - Re: One vs many directories

[Jean Louis]

Additionally, as a good example of faulty design, user is coerced to
ACCEPT local variables rather than is given fair choice.

As there is the option ! to "apply local variables and permanently
mark these values" but there is no option "not to apply local
variables and permanently mark these values".

That is not fair choice. It pushes user to finally ! apply and accept it, but
does not give chance to permanently ignore it.


Do you want to apply it?  You can type
y  -- to apply the local variables list.
n  -- to ignore the local variables list.
!  -- to apply the local variables list, and permanently mark these
      values (*) as safe (in the future, they will be set automatically.)

Re: Local variables insecurities - Re: One vs many directories

[Tom Gillespie]

> As there is the option ! to "apply local variables and permanently
> mark these values" but there is no option "not to apply local
> variables and permanently mark these values".

I have a longer reply that I will send tomorrow, but wanted to respond
to this.

Yes exactly! I have the equivalent complaint in the draft extras from
my previous message! I actually implemented a blacklist for file local
variables in orgstrap because it is a critical security feature. The
fact that it is missing is absolutely a bug and is extremely annoying
for some of my current workflows where I have local variables that I
never want to accept.

Re: Local variables insecurities - Re: One vs many directories

[Jean Louis]

* Tom Gillespie <tgbugs@gmail.com> [2020-11-26 09:19]:
> > As there is the option ! to "apply local variables and permanently
> > mark these values" but there is no option "not to apply local
> > variables and permanently mark these values".
> 
> I have a longer reply that I will send tomorrow, but wanted to respond
> to this.
> 
> Yes exactly! I have the equivalent complaint in the draft extras from
> my previous message! I actually implemented a blacklist for file local
> variables in orgstrap because it is a critical security feature. The
> fact that it is missing is absolutely a bug and is extremely annoying
> for some of my current workflows where I have local variables that I
> never want to accept.

Then maybe just add to the same bug your opinion.

Re: Local variables insecurities - Re: One vs many directories

[Detlef Steuer]

Am Thu, 26 Nov 2020 08:31:29 +0300
schrieb Jean Louis <bugs@gnu.support>:

> That is not fair choice. It pushes user to finally ! apply and accept
> it, but does not give chance to permanently ignore it.
> 
> 
> Do you want to apply it?  You can type
> y  -- to apply the local variables list.
> n  -- to ignore the local variables list.
> !  -- to apply the local variables list, and permanently mark these
>       values (*) as safe (in the future, they will be set
> automatically.)

Well, if you are arguing for a user who chose to use emacs, configured
mutt to invoke emacs etc,  but at the same time has no idea he chose a
mighty tool, as in has no clue about programming, variables etc., then
this choice won't help neither. If you have no idea about local
variables, you can not make an informed decision about *anything*
involving the concept of local variables. Besides stopping and reading
the manual.

May be I miss something, but then I miss it :-)

Furthermore this discussion has imho long left the main topic of this list.

Detlef

Re: Local variables insecurities - Re: One vs many directories

[Jean Louis]

* Detlef Steuer <steuer@hsu-hh.de> [2020-11-26 14:46]:
> Am Thu, 26 Nov 2020 08:31:29 +0300
> schrieb Jean Louis <bugs@gnu.support>:
> 
> > That is not fair choice. It pushes user to finally ! apply and accept
> > it, but does not give chance to permanently ignore it.
> > 
> > 
> > Do you want to apply it?  You can type
> > y  -- to apply the local variables list.
> > n  -- to ignore the local variables list.
> > !  -- to apply the local variables list, and permanently mark these
> >       values (*) as safe (in the future, they will be set
> > automatically.)
> 
> Well, if you are arguing for a user who chose to use emacs, configured
> mutt to invoke emacs etc

I do not. Any file opened any how by Emacs is invoking those variables. 

Please send your opinions to bug:
https://debbugs.gnu.org/cgi/bugreport.cgi?bug=44837 by writing to:
44837@debbugs.gnu.org

> .but at the same time has no idea he chose a
> mighty tool, as in has no clue about programming, variables etc.,

That is right, I have given references. Do not expect users who obtain
Emacs editor to be programmers. When user is asked anything, it should
be self-documenting and it should have references to that
documentation. That is why majority of actions with Emacs do have ?
for help at least. Invoking unsafe variables does not even involve any
help or references to manual.

>  then
> this choice won't help neither. If you have no idea about local
> variables, you can not make an informed decision about *anything*
> involving the concept of local variables. Besides stopping and reading
> the manual.

That is right, but that will be picked by those better positioned few.

Re: Local variables insecurities - Re: One vs many directories

[Greg Minshall]

Tom,

> 2. If mutt is launching Emacs, you can pass --eval "(setq
>    enable-local-eval nil)" on the command line and all file local
>    variables will be ignored and treated as plain text.

maybe that is one thing that could really help here.  possibly mutt and
other emacs-based mail readers, when putting up a received e-mail
message, should by default do just that (and, also, '(setq
enable-local-variables :safe)'?).

i use mh-e, and it does *not* seem to do that.  but, if emacs is niche,
mh-e is the niche-squared. :)

cheers, Greg

Re: Local variables insecurities - Re: One vs many directories

[Jean Louis]

* Greg Minshall <minshall@umich.edu> [2020-11-26 08:34]:
> Tom,
> 
> > 2. If mutt is launching Emacs, you can pass --eval "(setq
> >    enable-local-eval nil)" on the command line and all file local
> >    variables will be ignored and treated as plain text.
> 
> maybe that is one thing that could really help here.  possibly mutt and
> other emacs-based mail readers, when putting up a received e-mail
> message, should by default do just that (and, also, '(setq
> enable-local-variables :safe)'?).
> 
> i use mh-e, and it does *not* seem to do that.  but, if emacs is niche,
> mh-e is the niche-squared. :)

I have sent summary of our discussion here to the bug #44837 for
further review, we may switch to better Org related subjects here if
you people agree.

https://debbugs.gnu.org/cgi/bugreport.cgi?bug=44837

Re: Local variables insecurities - Re: One vs many directories

[Christian Moe]

Located Eric's message and can confirm the same for mu4e (no query about
setting/evaluating anything upon opening the message).

cm

Eric S Fraga writes:

> On Wednesday, 25 Nov 2020 at 15:38, Jean Louis wrote:
>> I have not configured anything. In fact I have opened the email and I
>> was surprised that I am getting those dialogues to execute local
>> variables.
>
> Very strange.  It was my email that instigated this part of the
> thread.  I can view my email and I do not get asked to set any locate
> variables or, in this case, evaluate the elisp expression.  Out of
> curiosity, what mail reader did you use that actually interprets file
> local variables?  Gnus article view does not.

Re: One vs many directories

[Dr. Arne Babenhauserheide]

[-- Attachment #1: Type: text/plain, Size: 1556 bytes --]


Jean Louis <bugs@gnu.support> writes:

> * Dr. Arne Babenhauserheide <arne_bab@web.de> [2020-11-24 21:51]:
>> 
>> Jean Louis <bugs@gnu.support> writes:
>> >> The start of the local variables list should be no more than 3000
>> >> > characters from the end of the file
>> >> 
>> >> 
>> >> Given the length of the email, I guess this is why Emacs saw the variables
>> >> as being within the correct range.
>> >
>> > Yes thank you. I was thinking Emacs will do that only in files where
>> > it recognizes some comments or no comments and that variables need
>> > to be pretty down in the file, on the bottom. Now I learn it is not
>> > so.
>> >
>> > That is security issue.
>> 
>> Why is it a security issue? The variables do need to be close to the end
>> — 3000 characters is only about 50 lines.
>
> Emacs users, Org users on our mailing lists are not so private. Their
> names and email addresses are in the public database. Spammer can
> construct phishing type of an email, including something like Org news
> or something and send such email to users. Among let us say 3000
> people there will be percentage of users that will say Y to invoke the
> local variables due to lack of knowing what is it doing to computer.

That isn’t what I meant with my question. What I meant is: Why is it a
security issue that the variable cannot only be at the exact end of the
file but instead can be in the last 3000 characters of the file?

Best wishes,
Arne
-- 
Unpolitisch sein
heißt politisch sein
ohne es zu merken

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 1125 bytes --]

Local variables liberties

[Jean Louis]

* Dr. Arne Babenhauserheide <arne_bab@web.de> [2020-11-25 11:11]:
> 
> Jean Louis <bugs@gnu.support> writes:
> 
> > * Dr. Arne Babenhauserheide <arne_bab@web.de> [2020-11-24 21:51]:
> >> 
> >> Jean Louis <bugs@gnu.support> writes:
> >> >> The start of the local variables list should be no more than 3000
> >> >> > characters from the end of the file
> >> >> 
> >> >> 
> >> >> Given the length of the email, I guess this is why Emacs saw the variables
> >> >> as being within the correct range.
> >> >
> >> > Yes thank you. I was thinking Emacs will do that only in files where
> >> > it recognizes some comments or no comments and that variables need
> >> > to be pretty down in the file, on the bottom. Now I learn it is not
> >> > so.
> >> >
> >> > That is security issue.
> >> 
> >> Why is it a security issue? The variables do need to be close to the end
> >> — 3000 characters is only about 50 lines.
> >
> > Emacs users, Org users on our mailing lists are not so private. Their
> > names and email addresses are in the public database. Spammer can
> > construct phishing type of an email, including something like Org news
> > or something and send such email to users. Among let us say 3000
> > people there will be percentage of users that will say Y to invoke the
> > local variables due to lack of knowing what is it doing to computer.
> 
> That isn’t what I meant with my question. What I meant is: Why is it a
> security issue that the variable cannot only be at the exact end of the
> file but instead can be in the last 3000 characters of the file?

Not that I really answered on that meaning. In that meaning I find it
troublesome that Local variables will work by formatting of any kind

Example would be when there are OTHER things intertwined with local
variables. The file with this text below still works, and it is not
necessary that it works in that way. My eye goes to the end of
file. My expectation is not that local variables work anywhere in the
file just by counting number of characters as such ends of files can
look any how, formatting is too free and I would make it rigid that
users can look with their eyes easily to where those local variables
really are.

I would rather support that Local variables cannot start in the column
like 10, maybe they can start in column up to 5th or something similar
and that their End: cannot be longer then few lines from the bottom
and to be highlighted by default.


Some file

Lorem ipsum dolor sit amet, consectetur adipiscing elit. Donec a diam
lectus. Sed sit amet ipsum mauris. Maecenas congue ligula ac quam
viverra nec consectetur ante hendrerit. Donec et mollis
dolor. Praesent et diam eget libero egestas mattis sit amet vitae
augue. Nam tincidunt congue enim, ut porta lorem lacinia
consectetur. Donec ut libero sed arcu vehicula ultricies a non                                                                      Local Variables:
consectetur. Donec ut libero sed arcu vehicula ultricies a non                                                                      ivy-mode: 1
consectetur. Donec ut libero sed arcu vehicula ultricies a non                                                                      End:
tortor. Lorem ipsum dolor sit amet, consectetur adipiscing                                                                          ivy-mode: 1
elit. Aenean ut gravida lorem. Ut turpis felis, pulvinar a semper sed,                                                              End:
adipiscing id dolor. Pellentesque auctor nisi id magna consequat                      
sagittis. Curabitur dapibus enim sit amet elit pharetra tincidunt
feugiat nisl imperdiet. Ut convallis libero in urna ultrices
accumsan. Donec sed odio eros. Donec viverra mi quis quam pulvinar at
malesuada arcu rhoncus. Cum sociis natoque penatibus et magnis dis
parturient montes, nascetur ridiculus mus. In rutrum accumsan
ultricies. Mauris vitae nisi at sem facilisis semper ac in est.

Re: One vs many directories

[Tom Gillespie]

> > That is security issue.
>
> Why is it a security issue? The variables do need to be close to the end
> — 3000 characters is only about 50 lines.

It isn't a security issue by itself. Emacs never automatically runs
eval file local variables unless you have tampered with
enable-local-eval, in which case the tamperin is the security issue
not the existence of the local variables list.

Thus it is only a security issue if you permanently accept that eval
file local variable and then open random org files that use it with a
malicious startup block. An eval file local variable like that which
blindly executes an org babel block should never be permanently
accepted

Best,
Tom

Re: One vs many directories

[Tim Cross]

Tom Gillespie <tgbugs@gmail.com> writes:

>> > That is security issue.
>>
>> Why is it a security issue? The variables do need to be close to the end
>> — 3000 characters is only about 50 lines.
>
> It isn't a security issue by itself. Emacs never automatically runs
> eval file local variables unless you have tampered with
> enable-local-eval, in which case the tamperin is the security issue
> not the existence of the local variables list.
>
> Thus it is only a security issue if you permanently accept that eval
> file local variable and then open random org files that use it with a
> malicious startup block. An eval file local variable like that which
> blindly executes an org babel block should never be permanently
> accepted
>

Quite right Tom.

If people are really concerned about security, they should look first at
their use of repositories like MELPA. There is no formal review or
analysis of packages in these repositories, yet people will happily
select some package and install it.


--
Tim Cross

Re: One vs many directories

[Jean Louis]

* Tim Cross <theophilusx@gmail.com> [2020-11-24 23:40]:
> > Thus it is only a security issue if you permanently accept that eval
> > file local variable and then open random org files that use it with a
> > malicious startup block. An eval file local variable like that which
> > blindly executes an org babel block should never be permanently
> > accepted
> >
> 
> Quite right Tom.
> 
> If people are really concerned about security, they should look first at
> their use of repositories like MELPA. There is no formal review or
> analysis of packages in these repositories, yet people will happily
> select some package and install it.

That is analogous to enabling local variables because user has been
asked. Popping up a window with question is often a dialogue that
users are asked in other software. Dialogues are often not read, just
as I was not reading it for years and I did click YES many times.

Using such variables is unsafe and the default should be not to
execute it without any question. Only when user enables local
variables then user should be asked to execute it. That would mean
that aware user knows why that is needed. Such will be able to answer
questions YES or NO.

Unaware users must answer something. To be aware one has to know Emacs
Lisp and deeper functions of Emacs.  In beginning years it was just
fine to assume so due to general computing interests and people being
interested in every detail, today there are even more users of Emacs
who will not know what is going on.

I do not know for you, but when computer asks me anything YES or NO,
my tendency is to answer YES regardless if I have read it or not. This
same tendency may be with thousands of other users.

If I have invoked something on computer and I get asked anything, I
have tendency to approve whatever comes on me as I approved it by
invoking some action. Not that I am doing it every time yet I have the
tendency of doing it.

Observing users who are asked questions upon invokation of other
software I can say that many times users just click one of the
options, either YES or NO, but without real regard to the
meanings. The purpose to click either YES or NO is to continue one
step forward and randomity decides later what happens.

Re: One vs many directories

[Tim Cross]

Jean Louis <bugs@gnu.support> writes:

> Observing users who are asked questions upon invokation of other
> software I can say that many times users just click one of the
> options, either YES or NO, but without real regard to the
> meanings. The purpose to click either YES or NO is to continue one
> step forward and randomity decides later what happens.

You cannot prevent people from making bad decisions. Saying yes to
something when you don't know what it means is like using the same
weak password for everything. There has been massive amounts of
communication and education out there to let people know about the
risks. If they choose to ignore it or follow practices which are unsafe,
it is their tough luck.

We need to encourage people to take more responsibility for their
actions, not less. One important component of this is allowing the
consequences for bad decisions to occur and not spend endless resources
protecting people from themselves.

If your asked to do something and your clearly told that doing so might
be unsafe and your given an option not to do it which is just as easy to
perform and you still decide to do it, that decision is on you.

The alternative is to remove extremely useful functionality from many
responsible users because of an unknown number of others who make poor
decisions.

Furthermore, keep in mind that this ability in Emacs has been around for
longer than many users have been alive. I've been using it for nearly 30
years and have participated in many forums over that time. I have yet to
hear of a single security incident occurring because of local variables.
That doesn't mean such incidents have not occurred, but it does likely
mean they are rare.

--
Tim Cross

Local variables issue - Re: One vs many directories

[Jean Louis]

* Tim Cross <theophilusx@gmail.com> [2020-11-25 08:54]:
> 
> Jean Louis <bugs@gnu.support> writes:
> 
> > Observing users who are asked questions upon invokation of other
> > software I can say that many times users just click one of the
> > options, either YES or NO, but without real regard to the
> > meanings. The purpose to click either YES or NO is to continue one
> > step forward and randomity decides later what happens.
> 
> You cannot prevent people from making bad decisions. Saying yes to
> something when you don't know what it means is like using the same
> weak password for everything. There has been massive amounts of
> communication and education out there to let people know about the
> risks. If they choose to ignore it or follow practices which are unsafe,
> it is their tough luck.

I do agree only that it is too general to apply here. That is specific
case of showing a dialogue with potential dangers to users who did not
specify those local variables and probably do not need such!

If you personally specify local variables you need them and know what
it is. That is fine.

It is general design that was meant for hackers in beginning stages of
Emacs development. Today many people use Emacs who may not be
hackers. Subset of those people will say YES to anything.

It is possible to prevent people to make bad decisions and it is easy,
simply don't ask them with the option to make bad decision. Disabling
local variables by default would be better decision.

I think that it is not possible today to change the default. Design is
flawed. From a view point of text editor should not ask user to
execute anything by default. User should enable "Local variables
detection" when user wish to get asked about executions.

> We need to encourage people to take more responsibility for their
> actions, not less.

I do fully agree on that statement, only it is too general and not
specific. I have presented my specific case how I have been answering
YES to that dialogue by thinking it is something necessary to read the
file properly. I had misunderstandings. Since some time I have the
general rule to answer NO on such dialogues. Would there be some
malicious intention in those years the intruder would be quite
successful.

One could fetch the external program and call it "general Emacs
enhancement" and open up a backdoor shell into the system.

To encourage people to take more responsibility is not necessarily
general principle for GNU Emacs.

And to encourage them alone will never work. To gain any
responsibility person needs knowledge or information. Driving car
without knowledge is impossible.

Thus pushing some kinds of responsibilities to user, coercing user to
decide on something that user does not understand itself lacks
one part of responsibility.

If user is informed what are local variables or is using them already
or reads manual and understands dangers, then user has acquired
knowledge to be able to reason when such dialogue pops up with YES or
NO. In general the answer YES can ruin your data and computing, and
users are not aware of it.

When somebody is not aware of what is doing that is not condition of
encouragement, rather condition of lack of responsibility.

> One important component of this is allowing the consequences for bad
> decisions to occur and not spend endless resources protecting people
> from themselves.

The YES/NO dialogue was invented by programmers and not by users. So
it was never an initial decision of the user who reads file from other
person, to be asked in the first place if something in that text file
should get executed.

It may also negatively impact Emacs's image as software:
https://cve.mitre.org/cgi-bin/cvekey.cgi?keyword=emacs

> If your asked to do something and your clearly told that doing so might
> be unsafe and your given an option not to do it which is just as easy to
> perform and you still decide to do it, that decision is on you.

Problem is with the assumption that user is "clearly told". I have
used Emacs since 1999 and I was all the time in scratch buffers and
did not really know it is for Emacs Lisp. I have been putting my notes
there. And I have ignored many functions of Emacs and used it to
program in other programming languages. Despite that I did read the
manual I did not clearly get information that there is programming
language built-in and it could execute any code. Despite knowing about
packages and installing them I did not know how it works. That is my
personal reality. I was playing tetris and I did not know it is
program that is loaded and executed. For me that was a built-in
feature, something that Richard Stallman and others invented and
built-in.

> The alternative is to remove extremely useful functionality from
> many responsible users because of an unknown number of others who
> make poor decisions.

Review that statement of yours again as you said what I am saying:
unknown number of others who make poor decisions.

Functionality need not be removed from Emacs to have or to design more
safer computing environment.

I have just opened new.txt file and used M-x
add-file-local-variable-prop-line only to verify how dialogue really
looks like:

The local variables list in new.txt
contains values that may not be safe (*).

Do you want to apply it?  You can type
y  -- to apply the local variables list.
n  -- to ignore the local variables list.
!  -- to apply the local variables list, and permanently mark these
      values (*) as safe (in the future, they will be set automatically.)

  * ivy-mode : 1

Then I switch from my programmers view point to let us say translator
view point. Translator who is introduced to Emacs as excellent text
editor and has to translate from German language to some other
language. Such translator would go through the Emacs tutorial and
learn how to open files, save files, use the keys. Translator cannot
possibly be expected to take any responsibilities for this dialogue as
translator that I know would never understand or be able to understand
what are these terms:

- "local" as in "local variables" would not be clear what it means local
  as there is no context that is understandable. Hacker and
  experienced user can understand it. Translator cannot. There is only
  general definition of "local" and that one does not apply to this
  context, one has to read the Emacs Manual fully to understand it and
  that is not what many people do when editing text.

- "variables" is programming term that translator is faced with and
  cannot possibly know what is meant with it. All what translator
  knows is that he go the file from other person and is now faced with
  the dialogue. There is no way for such person to responsibly make
  any decision, neither YES or NO. Translator will choose one of them
  to pass through the process, not to make the decision.

- "values" from the view point of translator is vague and cannot be
  understood. What values? Where?

- "may not be safe (*)" -- while symbol * may be used as reference
  this also may not be clear. Then what is "safe"? It is not clear
  from the dialogue context how that cannot be safe. Translator is
  sitting on computer and have been using typewriter and other
  computers for whole his life and will say "I was always safe" so why
  should it not be safe for me. And out of misunderstanding or protest
  will say YES to it.

- "Do you want to apply it?" -- to translator this is vague, as
  translator is not programmer but is faced with programming
  decisions that translator cannot understand. If translator wanted to
  open the file this may be understood as further approval to open the
  file. So if translator is opening the file, why would not translator
  like to apply it? Sure translator wants to apply it, damn YES!

- "to ignore the local variables list" -- means as well little and
  nothing to translator. Finally all that translator wanted is to open
  up text editor to translate the legal document hanging in front of
  him from German language to his local language and to earn some
  money. Such person may not be inclined to ignore things, damn YES!

- "!  -- to apply the local variables list, and permanently mark
  these" -- after several botherings translator may be inclined to not
  get bothered again, so let me ! do that, fine!

And after many such attempts translator may get bothered with the
dialogue that in the ends answers YES on any such dialogue.

I do understand that feature is built-in and files depend on it by
default. And while I support more safer solution, I do not support
that it should be removed as a default feature as it is for decades
late to design it safer.

But then this dialogue can be improved as it is very clear that
dialogue is handling possible security issue:
  
The local variables list in new.txt
contains values that may not be safe (*).

Do you want to apply it?  You can type
y  -- to apply the local variables list.
n  -- to ignore the local variables list.
!  -- to apply the local variables list, and permanently mark these
      values (*) as safe (in the future, they will be set automatically.)

  * ivy-mode : 1

Some words there can be hyperlinks to give knowledge to the user on
what is actually user deciding about, so various terms in the dialogue
can be hyperlinks to various parts in the manual. That way user may
get the real and straight access to information to make an informed
decision rather being left alone and called not responsible.

> Furthermore, keep in mind that this ability in Emacs has been around
> for longer than many users have been alive.

Yes, and that does not make it safer. My opinion is that it was bad
design for general public. Before 30 years there was not that many
users and people were in general much more interested in computing and
active computer use such as programming. Today the interest goes more
towards entertainment and passive computer use as that is what largest
corporations promote. The number of programmers did not increase
proportionally with the number of computers and computer users. We are
getting dumber as people, not smarter.

> I've been using it for nearly 30 years and have participated in many
> forums over that time. I have yet to hear of a single security
> incident occurring because of local variables.  That doesn't mean
> such incidents have not occurred, but it does likely mean they are
> rare.

That is hardly indication. When design is bad and something happens to
the user only those experienced and knowledgable will be able to reach
the forum or mailing list to report something like that.

If user is not knowledgable about local variables and already press
YES when it should be NO, then such user will never reach to report it
to you.

End of 1998 proprietary OS on my computer freezed. There was nothing
special going on, it just freezed and I had no other option but to
reset it. After the reset I could enter the system without any
problems or warnings and there was no file any more. While OS did
function files were not there and it was not nice to lose
files. Neither I have went to Internet, neither called anybody to
complain about that, just nothing, I was faced with condition outside
of control and did not know there could be helped to it. This would
happen to those who are affected by security flaws.

Re: One vs many directories

[Jean Louis]

* Tim Cross <theophilusx@gmail.com> [2020-11-24 23:40]:
> If people are really concerned about security, they should look first at
> their use of repositories like MELPA. There is no formal review or
> analysis of packages in these repositories, yet people will happily
> select some package and install it.

Interesting that you are one who mentions that. There are just few
people ever mentioned it.

I am still in process of the review of MELPA packages and its
system. There are many security issues.

Package signing is one example. It does not offer much of security
when packages are signed automatically, but it raises level of
security.

MELPA packages and archive-contents are not PGP signed, while GNU ELPA
packages are signed.

Licensing issues are also a problem with MELPA as it becomes unclear
if I have got the license or not when author does not have a proper
name. It is not relevant if majority of people do not think or are not
aware of licensing as I have to think of it for software that I may
re-use, distribute, modify. Did I really get the license if user is
named "nick-abc" and have no possible contact information? In some
cases for subset of MELPA packages there is no way to verify who
really wrote piece of software and if I have received the license
legally. Due diligence is on my side. I cannot just claim "But he gave
me license" will not help if I have not done proper due diligence,
court would not be on my side.

Other issue is that MELPA philosophy is to accept any kind of software
even if software has been made to drive or control proprietary
software.

For that reason there is now non-GNU ELPA being developed where useful
packages will be distributed from.

Re: One vs many directories

[Tim Cross]

Jean Louis <bugs@gnu.support> writes:

> * Tim Cross <theophilusx@gmail.com> [2020-11-24 23:40]:
>> If people are really concerned about security, they should look first at
>> their use of repositories like MELPA. There is no formal review or
>> analysis of packages in these repositories, yet people will happily
>> select some package and install it.
>
> Interesting that you are one who mentions that. There are just few
> people ever mentioned it.
>
> I am still in process of the review of MELPA packages and its
> system. There are many security issues.
>
> Package signing is one example. It does not offer much of security
> when packages are signed automatically, but it raises level of
> security.
>
> MELPA packages and archive-contents are not PGP signed, while GNU ELPA
> packages are signed.
>

IMO signing of packages is irrelevant when there is no formal review
process or even any formal process to verify the credentials of
signatures. In fact, just adding signing would likely be
coutner-productive as it would give the impression of some sort of
security where there is none.

Basically, anyone can upload anything to MELPA. The only way anyone
would find out that an uploaded package has malicious code is if someone
does a code review and spots the malicious payload. Even once they find
that, there is little chance of being able to attribute the actions to
any individual because no real identity vetting is conducted. MELPA is
the wild west.

The new non-GNU repository has bene setup precisely due to both the
licensing issue and the fact many MELPA packages recommend/encourage the
use of non-free software/services. While non-GNU will improve this
situation, I don't believe there are any plans to actively review the
code in the packages. So, like MELPA, all you really have to go on is
package reputation. You cannot have any high level of confidence a
package does not contain malicious code other than an expectation that
if it is used by a sufficiently large enough number of users, it is
unlikely.

this is not an issue unique to Emacs. You only have to look at the
issues both Google's play store and Apples app store have had in the
past to see what the risks are. Both Google and Apple have put large
amounts of resources into trying to ensure their repository content is
safe and yet they still have failures. Something like GNU Emacs has
nowhere near the same resources, so is unlikely to come even close to
the same level of security.

Security issues in Emacs packages

[Jean Louis]

* Tim Cross <theophilusx@gmail.com> [2020-11-25 10:01]:
> 
> Jean Louis <bugs@gnu.support> writes:
> 
> > * Tim Cross <theophilusx@gmail.com> [2020-11-24 23:40]:
> >> If people are really concerned about security, they should look first at
> >> their use of repositories like MELPA. There is no formal review or
> >> analysis of packages in these repositories, yet people will happily
> >> select some package and install it.
> >
> > Interesting that you are one who mentions that. There are just few
> > people ever mentioned it.
> >
> > I am still in process of the review of MELPA packages and its
> > system. There are many security issues.
> >
> > Package signing is one example. It does not offer much of security
> > when packages are signed automatically, but it raises level of
> > security.
> >
> > MELPA packages and archive-contents are not PGP signed, while GNU ELPA
> > packages are signed.
> >
> 
> IMO signing of packages is irrelevant when there is no formal review
> process or even any formal process to verify the credentials of
> signatures. In fact, just adding signing would likely be
> coutner-productive as it would give the impression of some sort of
> security where there is none.

When user receives a signed package from GNU ELPA, that means it comes
from GNU ELPA. If signed package is then distributed by mirrors or
other websites users who enable package signature verifications will
know that package still comes from GNU ELPA, and not from Chinese
distributor. If the package is tampered the signature verification
will not allow installation of the package.

Variable package-check-signature is not enabled by default and it is
due to similar issue just as local variables.

Users who seek to verify the credential of the signatures can do so in
human non-automated way as that is also how GnuPG instructions advise
users to do so. The GPG/PGP keys are in ~/.emacs.d/elpa/gnupg/ and
users can contact GNU ELPA maintainers and verify the signature. M-x
package-install will automatically verify signatures if the option
package-check-signature has been enabled.

That it does add to security shows the fact that GNU/Linux
distributions do sign packages. There is difference if the package
comes from trusted source or not trusted source. 

> Basically, anyone can upload anything to MELPA.

Maintainers verifies the package initially for certain conventions,
after that, how I have understood it, packages are automatically
pulled from Git. The more authors give packages to MELPA the more
insecurity probability is raised.

GNU ELPA how I understand it, I may be wrong, works like this:

1) packages are uploaded to GNU ELPA
2) then automatically signed by GNU ELPA PGP keys
3) offered for distribution

The point number (1) is human, not automated. Author decides when is
the package ripe for distribution and what is "release".

Git repository is never release and not meant to be "release". Git is
for collaborative development and users are made blind that it is some
kind of release while it is not. One shall always assume that Git
repository contains development versions not ready for public.

MELPA pulls those packages, correct me if I am wrong, automatically
from Git repositories without regard if the package is actually
release. That does not align or respect the established Emacs
conventions how packages should be released, if they are multi files
they should be in .tar file otherwise .el and there are version
numbers that MELPA fiddles with and makes possible conflicts and
introduces confusions.

In GNU ELPA authors decide when package is release ready and when
it should be released.

In MELPA authors only apply for their packages to be pulled out
automatically and offered for distribution.

Both repositories could be compromised but probability is becoming
larger and larger that by automatic pulling of packages something
worse happens.

MELPA cannot know possibly who is behind authors who offer those
packages for distribution and who has access or who may do something
malicious.

Some new similar package like angry-police-captain could serve for
potential attacks.

#+TITLE: <2020-10-23 Fri 18:28>  WTF angry-police-captain
#+AUTHOR: Jean Louis
- This should scrap information from a third party unknown website and
  show it in minibuffer. Function does not work, and yes, it is just
  one function inside. Good example of nonsensical
  "packages". *Deleted*

While similar packages can be made for entertainment they can be also
used to track users and save data that should not be saved. Update to
this package would not be checked by MELPA, and users who have enabled
it would continue using it. And package could suddenly start doing
something else. Author of the package could know how many users are
using it as package is actually fetching from their website. By
fetching the information from website the website can know many things
about those users such as their locations, operating system and
versions, etc. and could invoke specific malicious stuff for those
specific users including send different information to users by their
different location or other attributes.

For that reason MELPA's automatic pulling of packages and race to
offer "large package repository" is rather by its design detrimental
for future. I hope it will change, but currently that is unlikely.

> The only way anyone would find out that an uploaded package has
> malicious code is if someone does a code review and spots the
> malicious payload. Even once they find that, there is little chance
> of being able to attribute the actions to any individual because no
> real identity vetting is conducted. MELPA is the wild west.

Thank you for saying your observation.

> The new non-GNU repository has bene setup precisely due to both the
> licensing issue and the fact many MELPA packages recommend/encourage the
> use of non-free software/services. While non-GNU will improve this
> situation, I don't believe there are any plans to actively review the
> code in the packages.

For my personal uses I am reviewing packages and constructing list of
required packages that I may use and subset of my group would use.

There also exist possibility of annotation of packages or one could
make a feedback by users so that more things can be discovered about
packages. 

> So, like MELPA, all you really have to go on is package
> reputation. You cannot have any high level of confidence a package
> does not contain malicious code other than an expectation that if it
> is used by a sufficiently large enough number of users, it is
> unlikely.

Interpreting statistics is not for everyone. It would be nice that
users give a human feedback which can be used for package reputation.

If one counts "download statistics" that is incorrect to be used for
reputation.

If let us say imaginary, a package about angry-police-captain would
contain some malicious code, then if user cannot differentiate what is
reputation and download, then number of 1000+ downloads would be quite
convincing to load the package.

Download number is now used for reputation as it is currently the only
attribute that may be obtained.

> this is not an issue unique to Emacs. You only have to look at the
> issues both Google's play store and Apples app store have had in the
> past to see what the risks are.

Of course, I know that. That is why there is no Google on any of my
mobile phones as they run LineageOS operating system. I would switch
to Replicant would I have access to supported hardware. All our
clients and staff members are warned not to use the Google.

Do you think MELPA authors would act on these comments to change
something?

Re: Security issues in Emacs packages

[tomas]

[-- Attachment #1: Type: text/plain, Size: 167 bytes --]

On Wed, Nov 25, 2020 at 11:23:27AM +0300, Jean Louis wrote:

[...]

> [...] and not from Chinese distributor [...]

I think this was an unnecessary slur.

Cheers
 - t

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: Security issues in Emacs packages

[Jean Louis]

* tomas@tuxteam.de <tomas@tuxteam.de> [2020-11-25 12:08]:
> On Wed, Nov 25, 2020 at 11:23:27AM +0300, Jean Louis wrote:
> 
> [...]
> 
> > [...] and not from Chinese distributor [...]
> 
> I think this was an unnecessary slur.

Why, there is legitimate mirror in China.

I did not mean nothing wrong with it. I hope nobody gets offended.

I was just thinking on mirrors like when using /etc/apt/sources.list
on Debian GNU/Linux and sources come from let us say "Digital Ocean"
servers. It is not official mirror of Debian, but if packages are
signed by Debian keys then I do not mind as I know they are not
tampered.

I hope I have expressed myself now more clear.

Re: Security issues in Emacs packages

[tomas]

[-- Attachment #1: Type: text/plain, Size: 800 bytes --]

On Wed, Nov 25, 2020 at 12:26:11PM +0300, Jean Louis wrote:
> * tomas@tuxteam.de <tomas@tuxteam.de> [2020-11-25 12:08]:
> > On Wed, Nov 25, 2020 at 11:23:27AM +0300, Jean Louis wrote:
> > 
> > [...]
> > 
> > > [...] and not from Chinese distributor [...]
> > 
> > I think this was an unnecessary slur.
> 
> Why, there is legitimate mirror in China.
> 
> I did not mean nothing wrong with it. I hope nobody gets offended.

I'm not. I don't assume (even suspect) bad intention at all.
And I don't want to make a state affair of it. I just wanted
to mirror the metaphor you used ("Chinese distributor" as
"untrusted instance") which seems somewhat problematic. We
all do this, myself not the least. I'm happy whenever someone
points that out to me.

That's all :-)

Cheers
 - t

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: Security issues in Emacs packages

[Tim Cross]

Jean Louis <bugs@gnu.support> writes:

> * Tim Cross <theophilusx@gmail.com> [2020-11-25 10:01]:
>>
>> Jean Louis <bugs@gnu.support> writes:
>>
>> > * Tim Cross <theophilusx@gmail.com> [2020-11-24 23:40]:
>> >> If people are really concerned about security, they should look first at
>> >> their use of repositories like MELPA. There is no formal review or
>> >> analysis of packages in these repositories, yet people will happily
>> >> select some package and install it.
>> >
>> > Interesting that you are one who mentions that. There are just few
>> > people ever mentioned it.
>> >
>> > I am still in process of the review of MELPA packages and its
>> > system. There are many security issues.
>> >
>> > Package signing is one example. It does not offer much of security
>> > when packages are signed automatically, but it raises level of
>> > security.
>> >
>> > MELPA packages and archive-contents are not PGP signed, while GNU ELPA
>> > packages are signed.
>> >
>>
>> IMO signing of packages is irrelevant when there is no formal review
>> process or even any formal process to verify the credentials of
>> signatures. In fact, just adding signing would likely be
>> coutner-productive as it would give the impression of some sort of
>> security where there is none.
>
> When user receives a signed package from GNU ELPA, that means it comes
> from GNU ELPA. If signed package is then distributed by mirrors or
> other websites users who enable package signature verifications will
> know that package still comes from GNU ELPA, and not from Chinese
> distributor. If the package is tampered the signature verification
> will not allow installation of the package.
>

I think you missed my point. There is no benefit in MELPA adopting
signed packages because there is no formal code review and no vetting of
the individuals who submit the code. If you have no controls in place
over the contents of what is being signed, the value of the signatures
as a security measure is drastically reduced. Yes, the valid signature
may provide some assurances as to where the package originated, but that
means little if the contents could be anything.

The situation with ELPA is a little better because those who maintain
the code are required to assign legal copyright to GNU. However, I'm not
sure how much checking is done to verify the information in those
assignments. As far as I know, there is no formal code review. A number
of the Emacs developers do perform some informal review, but as we all
know from the issues with openssl, informal reviews provide little
assurance. This is not a criticism of GNU or emacs developers. The
amount of resources necessary to perform formal review is much larger
than the available resources. On the whole, the emacs user community
appears to be happy with the current situation. If they were not, it
would be on the community to step up and do something about it.

As it stands, the signing of ELPA packages only provides assurance that
they are packages assembled by GNU. These signatures do not provide any
real assurance regarding the content of the packages other than they are
GPL's and do not recommend or encourage the use of non-free software.


> That it does add to security shows the fact that GNU/Linux
> distributions do sign packages. There is difference if the package
> comes from trusted source or not trusted source.
>

The question is what level of trust should you assume. With ELPA, all
you can really trust is that the package has a GPL license and does not
recommend/require the use of non-free software. There is little trust
that the package does not do something malicious or includes code which
may compromise the user's security. to provide that level of assurance,
you would need formal code reviews, which is not feasible given
available resources. I think it is important users are aware of this
limitation. Furthermore, I ask the question "Does having signed packages
imply a level of expected assurance which is higher than it really is?"
In other words, do users expect that a package is completely safe just
because it was downloaded from an official GNU ELPA repository?

>> Basically, anyone can upload anything to MELPA.
>
> Maintainers verifies the package initially for certain conventions,
> after that, how I have understood it, packages are automatically
> pulled from Git. The more authors give packages to MELPA the more
> insecurity probability is raised.
>
> GNU ELPA how I understand it, I may be wrong, works like this:
>
> 1) packages are uploaded to GNU ELPA
> 2) then automatically signed by GNU ELPA PGP keys
> 3) offered for distribution
>
Last time I looked, ELPA also supported 'external' packages where the
data is retrieved from an external git repository. I think org is one
such package.

> The point number (1) is human, not automated. Author decides when is
> the package ripe for distribution and what is "release".
>
> Git repository is never release and not meant to be "release". Git is
> for collaborative development and users are made blind that it is some
> kind of release while it is not. One shall always assume that Git
> repository contains development versions not ready for public.
>

Why? This is not normal. Git repositories contain all versions, both
production and development. What is production and what is development
is managed through branches and tags. Anyone who wants can clone the
ELPA git repository.

> MELPA pulls those packages, correct me if I am wrong, automatically
> from Git repositories without regard if the package is actually
> release. That does not align or respect the established Emacs
> conventions how packages should be released, if they are multi files
> they should be in .tar file otherwise .el and there are version
> numbers that MELPA fiddles with and makes possible conflicts and
> introduces confusions.

this is wrong. In melpa you specify either a commit (SHA) or a branch or
both. The repository owner has control over this. MELPA doesn't just
pull data from the repository because there has bene an update. You can
configure things so that whenever data is committed to a release branch,
it is pulled, but this is under the control of the repository owner. It
isn't that different to ELPA where the maintainer will either push new
data to the ELPA repository (or ask someone with write permission to
pull it from their repository).

>
> In GNU ELPA authors decide when package is release ready and when
> it should be released.
>
> In MELPA authors only apply for their packages to be pulled out
> automatically and offered for distribution.
>

You imply authors do not have control over when new releases are made.
This is not the case. They have full control.

> Both repositories could be compromised but probability is becoming
> larger and larger that by automatic pulling of packages something
> worse happens.

The risks between the two systems are not significantly different
because neither system performs any formal review of the code. In some
respects, this is more of an issue for ELPA because there is likely a
higher expectation that the code from ELPA can be trusted more than the
code from MELPA. ELPA also has a scaling challenge. Currently, anyone
who has the ability to push data to ELPA for a package also has the
ability to push to any directory (package), not just the package they
are maintainer for. This means that access to write permission for the
ELPA repository needs to be restricted to trusted users, which in turn
places more pressure on those trusted users to manage requests to update
data. As the number of ELPA packages increases, either more people will
need to be trusted or more pressure placed on those who are. As you
increase the number of people with write access you also increase the
risks.


>
> MELPA cannot know possibly who is behind authors who offer those
> packages for distribution and who has access or who may do something
> malicious.
>

The situation with ELPA is not much better. Yes, the authors are
required to sign over copyright, but what does that really tell you
about the author. How much vetting is done to verify those copyright
assignments? How much vetting is done to verify the identities of those
people? More importantly, how much of the code is formally reviewed?

The assumption that because a package is from ELPA it is safe is wrong.
This is the danger - an expectation that because the package is from
ELPA it is more trustworthy than a package from MELPA. The only thing
you really know for certain about a package from ELPA is that it has a
GPL license and it does not recommend or require non-free software. Any
review of the code in the package is informal and not guaranteed to
occur after every update. The requirement to assign copyright and the
fact at least some informal review is performed does provide some level
of assurance you don't get from MELPa, but it is a mistake to assume
that just because a package comes from ELPA it is safe or does not
include any significant security issues.

> Some new similar package like angry-police-captain could serve for
> potential attacks.
>
> #+TITLE: <2020-10-23 Fri 18:28>  WTF angry-police-captain
> #+AUTHOR: Jean Louis
> - This should scrap information from a third party unknown website and
>   show it in minibuffer. Function does not work, and yes, it is just
>   one function inside. Good example of nonsensical
>   "packages". *Deleted*
>
> While similar packages can be made for entertainment they can be also
> used to track users and save data that should not be saved. Update to
> this package would not be checked by MELPA, and users who have enabled
> it would continue using it. And package could suddenly start doing
> something else. Author of the package could know how many users are
> using it as package is actually fetching from their website. By
> fetching the information from website the website can know many things
> about those users such as their locations, operating system and
> versions, etc. and could invoke specific malicious stuff for those
> specific users including send different information to users by their
> different location or other attributes.
>

and the same thing is possible with ELPA. You may need to be a little
mor subtle and you may need to play 'the long game', but there is
nothing inherent in the ELPA process which provides protection against
this other than the valuable and incredible dedication of Emacs
developers. The problem is, informal processes for code review are
notoriously unreliable. We just have to look at what happened in the
openssl project to see how badly things can go wrong despite all good
intentions. In fact, your own words demonstrate the issue. You have a
belief/expectation that ELPA packages are safer than MELPA packages
because they are from a source you trust. However, without any formal
review of the code in those packages, that level of trust may not be
warranted.

So how big a risk are ELPA packages in reality? This is a difficult
thing to quantify. Yes, I do think there is lower risk with ELPA than
MELPA because even informal review is better than no review. I also
think that if you wanted to introduce a malicious package into the Emacs
ecosystem you would follow the path of least resistance, which is MELPA.
I also think the risk and reward calculations make Emacs packages a
lower risk - the effort required to get a malicious package adopted and
the rewards such effort would provide just don't add up. There are far
more rewarding options out there. However, I do think people need to
install packages with caution, regardless of whether they are from ELPA
or MELPA or anywhere else.

> For that reason MELPA's automatic pulling of packages and race to
> offer "large package repository" is rather by its design detrimental
> for future. I hope it will change, but currently that is unlikely.
>

The automatic pulling is not the issue. As long as there is no formal
review of code in packages, any method used is vulnerable. Regardless of
the approach, at the end of the day your trusting the author/maintainer
will do the right thing. This is true for both MELPA and ELPA. It also
includes trusting the maintainer won't make a mistake and that they have
good operational security and are not themselves compromised.


<snip>

>> So, like MELPA, all you really have to go on is package
>> reputation. You cannot have any high level of confidence a package
>> does not contain malicious code other than an expectation that if it
>> is used by a sufficiently large enough number of users, it is
>> unlikely.
>
> Interpreting statistics is not for everyone. It would be nice that
> users give a human feedback which can be used for package reputation.
>
> If one counts "download statistics" that is incorrect to be used for
> reputation.
>
> If let us say imaginary, a package about angry-police-captain would
> contain some malicious code, then if user cannot differentiate what is
> reputation and download, then number of 1000+ downloads would be quite
> convincing to load the package.
>
> Download number is now used for reputation as it is currently the only
> attribute that may be obtained.
>

that is not what I meant by reputation and number of downloads is not
the metric I use. What I meant by reputation is what others write about
the package - what is discussed in forums, what comes up with a google
search etc. I maintain a JS package which has over 100k downloads per
week. This means nothing to me and I suspect to most users. The
reputation for the package is based on what is in the issue/bug tracker,
how quickly issues are addressed and how many other packages or systems
use the package.

I think we have exhausted this topic now. It really is something which
should be discussed on the emacs-devel list rather than an org list. I
do think people probably need to be more aware of the risks associated
with all emacs packages, regardless of source, but probably best in a
more general emacs forum rather than this list.

--
Tim Cross

Re: Security issues in Emacs packages

[Jean Louis]

* Tim Cross <theophilusx@gmail.com> [2020-11-26 01:47]:
> I think you missed my point. There is no benefit in MELPA adopting
> signed packages because there is no formal code review and no vetting of
> the individuals who submit the code.

Do you think it is really their reason? Or maybe you are developer in
MELPA?

There is still difference if package comes from MELPA or from third
party archive, definitely signing would say this package was signed by
MELPA and other package was not signed by trusted key, so who is that?
Is the MD5SUM same as original? It would give some initiative to
investigate.

Packages are not audited that is so. I think audit can be quick by
using grep for some dangerous commands, I have already found rm -rf as
example command in one of packages, not as malicious one. One could
search for (shell-command and verify such and similar functions.

> If you have no controls in place over the contents of what is being
> signed, the value of the signatures as a security measure is
> drastically reduced. Yes, the valid signature may provide some
> assurances as to where the package originated, but that means little
> if the contents could be anything.

What you explain is logical to me, users though need better
information. One big DANGER should be given to users.

> As it stands, the signing of ELPA packages only provides assurance that
> they are packages assembled by GNU. These signatures do not provide any
> real assurance regarding the content of the packages other than they are
> GPL's and do not recommend or encourage the use of non-free
> software.

There is no absolute. Signing says about origins. Mirrors are placed
anywhere in the world, including behind Internet. It is one way for
users to verify origin and if source has been changed.

> The question is what level of trust should you assume. With ELPA, all
> you can really trust is that the package has a GPL license and does not
> recommend/require the use of non-free software. There is little trust
> that the package does not do something malicious or includes code which
> may compromise the user's security. to provide that level of assurance,
> you would need formal code reviews, which is not feasible given
> available resources.

Last month I could see that several packages were here and there
improved by developers so they do look into code and how much they do
I cannot know.

> I think it is important users are aware of this
> limitation. Furthermore, I ask the question "Does having signed
> packages imply a level of expected assurance which is higher than it
> really is?"  In other words, do users expect that a package is
> completely safe just because it was downloaded from an official GNU
> ELPA repository?

Download numbers on MELPA tells me that answer should be rather
positive, any package is safe to be installed. See
numbers. Information is no enough to teach users. More attention is
necessary. 

> Last time I looked, ELPA also supported 'external' packages where the
> data is retrieved from an external git repository. I think org is one
> such package.

Majority of GNU ELPA packages are external how I know about it, but
authors decide WHEN to upload them.

> > The point number (1) is human, not automated. Author decides when is
> > the package ripe for distribution and what is "release".
> >
> > Git repository is never release and not meant to be "release". Git is
> > for collaborative development and users are made blind that it is some
> > kind of release while it is not. One shall always assume that Git
> > repository contains development versions not ready for public.
> >
> 
> Why? This is not normal. Git repositories contain all versions, both
> production and development. What is production and what is development
> is managed through branches and tags. Anyone who wants can clone the
> ELPA git repository.

How I see practically, people hack on git master branches and main
branch need not be considered release ready. Git hosting websites then
have special section for releases. Git branch is not a release
according to what I know, it is revision control system or version
control system. Git often looks pretty different than release as
package. Of course everybody can clone. Point is that software is no
ripe. Maybe somebody else knows if Git can tell that software is ripe,
what I know it is not so. Author has to say when it is ripe for
release. 

> > MELPA pulls those packages, correct me if I am wrong, automatically
> > from Git repositories without regard if the package is actually
> > release. That does not align or respect the established Emacs
> > conventions how packages should be released, if they are multi files
> > they should be in .tar file otherwise .el and there are version
> > numbers that MELPA fiddles with and makes possible conflicts and
> > introduces confusions.
> 
> this is wrong. In melpa you specify either a commit (SHA) or a branch or
> both. The repository owner has control over this. MELPA doesn't just
> pull data from the repository because there has bene an update. You can
> configure things so that whenever data is committed to a release branch,
> it is pulled, but this is under the control of the repository owner. It
> isn't that different to ELPA where the maintainer will either push new
> data to the ELPA repository (or ask someone with write permission to
> pull it from their repository).

OK it is great that it is so. Are you maybe author doing it? Is there
any reference that authors are doing so? I have MELPA downloaded you
could tell me how do I see that author is deciding if package is for
release?

> You imply authors do not have control over when new releases are made.
> This is not the case. They have full control.

Sure they have for themselves. Do they have it for MELPA?

> The situation with ELPA is not much better. Yes, the authors are
> required to sign over copyright, but what does that really tell you
> about the author. How much vetting is done to verify those copyright
> assignments? How much vetting is done to verify the identities of those
> people? More importantly, how much of the code is formally reviewed?

Valid questions!

> The assumption that because a package is from ELPA it is safe is
> wrong.

Safer, not safe.

Assumption by majority is that any packages from anywhere are
safe. Downloads prove it.

> So how big a risk are ELPA packages in reality? This is a difficult
> thing to quantify. Yes, I do think there is lower risk with ELPA than
> MELPA because even informal review is better than no review.

Side note: ELPA is protocol, GNU ELPA is repository. MELPA ELPA should
be rather more correct name.

I can see all those points and I would like there is better code review.

> > For that reason MELPA's automatic pulling of packages and race to
> > offer "large package repository" is rather by its design detrimental
> > for future. I hope it will change, but currently that is unlikely.
> >
> The automatic pulling is not the issue. As long as there is no formal
> review of code in packages, any method used is vulnerable.

So is there automatic pulling?

I compare automatic pulling and building to author's decision on when
a package would be issued.

Jean

Re: Security issues in Emacs packages

[Tim Cross]

Jean Louis <bugs@gnu.support> writes:

>>
>> this is wrong. In melpa you specify either a commit (SHA) or a branch or
>> both. The repository owner has control over this. MELPA doesn't just
>> pull data from the repository because there has bene an update. You can
>> configure things so that whenever data is committed to a release branch,
>> it is pulled, but this is under the control of the repository owner. It
>> isn't that different to ELPA where the maintainer will either push new
>> data to the ELPA repository (or ask someone with write permission to
>> pull it from their repository).
>
> OK it is great that it is so. Are you maybe author doing it? Is there
> any reference that authors are doing so? I have MELPA downloaded you
> could tell me how do I see that author is deciding if package is for
> release?
>

You can clone the melpa repository and see the recipes for each package.

It depends on how the author specifies their MELPA recipe. They can
define their recipe based on a specific commit (SHA). If they do this,
it doesn't matter how often or when MELPA pulls from the repository as
they will always get the same commit.

They can also specify a branch rather than a commit SHA. In this case,
MELPA will retrieve updates from that branch, so when that branch is
updated, it will pull new data. In this case, it is up to the developer
to manage their 'release' branch appropriately. when they are ready for
a new release, they push their updates to the release branch and update
the version tag. This is pretty much the same as ELPA works for external
packages (those which don't manage their code within the GNU ELPA repository itself)


>
> So is there automatic pulling?
>
> I compare automatic pulling and building to author's decision on when
> a package would be issued.
>

Your model is flawed. You can have both automatic pulling AND author
control over when a new package is issued.

If author defines their MELPA recipe to use a SHA a new package will not
be issued until they update their recipe with a new SHA.

If author defines their MELPA recipe to pull from a release branch, a
new package will not be issued until they update the release branch and
version tag.

MELPA does not automatically generate a new package just because
something has changed within the git repository. It has to be a change
to a specified branch and update to the version tag or it has to be a
change in the recipe with an update to the commit SHA.

--
Tim Cross

Re: Security issues in Emacs packages

[Jean Louis]

* Tim Cross <theophilusx@gmail.com> [2020-11-26 02:40]:
> > OK it is great that it is so. Are you maybe author doing it? Is there
> > any reference that authors are doing so? I have MELPA downloaded you
> > could tell me how do I see that author is deciding if package is for
> > release?
> >
> 
> You can clone the melpa repository and see the recipes for each
> package.

I did before some time.

> It depends on how the author specifies their MELPA recipe. They can
> define their recipe based on a specific commit (SHA). If they do this,
> it doesn't matter how often or when MELPA pulls from the repository as
> they will always get the same commit.

I have not seen that, and I have assumed you would know better and
wanted to see how authors are reporting that package is ready for
release and I do not see that.

Recipes are like this:

(0blayout :repo "etu/0blayout-mode" :fetcher github)

(0x0 :url "https://git.sr.ht/~zge/nullpointer-emacs" :fetcher git)

(0xc :fetcher github :repo "AdamNiederer/0xc")

So that recipe alone does not tell me that author reports that new
package is ready, it is fetched from git, but there are parts of code
that I did not see that is why I am assuming you know it better.

> Your model is flawed. You can have both automatic pulling AND author
> control over when a new package is issued.

To make it practical tell me where is that author's control?

I have quick view of files and any recipe files in directory
melpa/recipes do not give me any pointers, it is all automated and
fetched from git.

> If author defines their MELPA recipe to use a SHA a new package will not
> be issued until they update their recipe with a new SHA.

You seem to be very confident and for this reason I assume you know it
better, but due to contradictions please show one practical recipe or
package where author has control on when is package ready to be
released.

$ grep sha *

on recipes does not give any reference.

$ grep commit *

eval-in-repl:              :commit  "origin/master")
git-auto-commit-mode:(git-auto-commit-mode :fetcher github :repo "ryuslash/git-auto-commit-mode")
git-commit:(git-commit :fetcher github
git-commit:            :files ("lisp/git-commit.el")
git-commit:            :old-names (git-commit-mode))
git-commit-insert-issue:(git-commit-insert-issue :fetcher gitlab :repo "emacs-stuff/git-commit-insert-issue")
vc-auto-commit:(vc-auto-commit :fetcher github :repo "thisirs/vc-auto-commit")
what-the-commit:(what-the-commit :fetcher github
what-the-commit:         :repo "danielbarbarito/what-the-commit.el")

So there is nothing I can find that points or references to what you
say.

> If author defines their MELPA recipe to pull from a release branch, a
> new package will not be issued until they update the release branch and
> version tag.

I am sorry I do not see reference to it. You are convincing but I do
not see reference.

Recipe for bar-cursor:
(bar-cursor :repo "ajsquared/bar-cursor"
:fetcher github)

Recipe for magit:

(magit :fetcher github
       :repo "magit/magit"
       :files ("lisp/magit"
               "lisp/magit*.el"
               "lisp/git-rebase.el"
               "Documentation/magit.texi"
               "Documentation/AUTHORS.md"
               "LICENSE"
	       (:exclude "lisp/magit-libgit.el"
			 ;; Cannot remove this yet because it would
			 ;; also be removed from the stable version.
			 ;; "lisp/magit-section.el"
			 )))

Repo magit/magit:
https://github.com/magit/magit

I have given you references, maybe I cannot read that well, so you can
give me references to show if authors have participation in decision.

Jean

Re: Security issues in Emacs packages

[Tim Cross]

Jean Louis <bugs@gnu.support> writes:

> * Tim Cross <theophilusx@gmail.com> [2020-11-26 02:40]:
>> > OK it is great that it is so. Are you maybe author doing it? Is there
>> > any reference that authors are doing so? I have MELPA downloaded you
>> > could tell me how do I see that author is deciding if package is for
>> > release?
>> >
>>
>> You can clone the melpa repository and see the recipes for each
>> package.
>
> I did before some time.
>
>> It depends on how the author specifies their MELPA recipe. They can
>> define their recipe based on a specific commit (SHA). If they do this,
>> it doesn't matter how often or when MELPA pulls from the repository as
>> they will always get the same commit.
>
> I have not seen that, and I have assumed you would know better and
> wanted to see how authors are reporting that package is ready for
> release and I do not see that.
>
> Recipes are like this:
>
> (0blayout :repo "etu/0blayout-mode" :fetcher github)
>
> (0x0 :url "https://git.sr.ht/~zge/nullpointer-emacs" :fetcher git)
>
> (0xc :fetcher github :repo "AdamNiederer/0xc")
>
> So that recipe alone does not tell me that author reports that new
> package is ready, it is fetched from git, but there are parts of code
> that I did not see that is why I am assuming you know it better.
>
>> Your model is flawed. You can have both automatic pulling AND author
>> control over when a new package is issued.
>
> To make it practical tell me where is that author's control?
>
> I have quick view of files and any recipe files in directory
> melpa/recipes do not give me any pointers, it is all automated and
> fetched from git.
>
>> If author defines their MELPA recipe to use a SHA a new package will not
>> be issued until they update their recipe with a new SHA.
>
> You seem to be very confident and for this reason I assume you know it
> better, but due to contradictions please show one practical recipe or
> package where author has control on when is package ready to be
> released.
>
> $ grep sha *
>
> on recipes does not give any reference.
>
> $ grep commit *
>
> eval-in-repl:              :commit  "origin/master")
> git-auto-commit-mode:(git-auto-commit-mode :fetcher github :repo "ryuslash/git-auto-commit-mode")
> git-commit:(git-commit :fetcher github
> git-commit:            :files ("lisp/git-commit.el")
> git-commit:            :old-names (git-commit-mode))
> git-commit-insert-issue:(git-commit-insert-issue :fetcher gitlab :repo "emacs-stuff/git-commit-insert-issue")
> vc-auto-commit:(vc-auto-commit :fetcher github :repo "thisirs/vc-auto-commit")
> what-the-commit:(what-the-commit :fetcher github
> what-the-commit:         :repo "danielbarbarito/what-the-commit.el")
>
> So there is nothing I can find that points or references to what you
> say.
>
>> If author defines their MELPA recipe to pull from a release branch, a
>> new package will not be issued until they update the release branch and
>> version tag.
>
> I am sorry I do not see reference to it. You are convincing but I do
> not see reference.
>
> Recipe for bar-cursor:
> (bar-cursor :repo "ajsquared/bar-cursor"
> :fetcher github)
>
> Recipe for magit:
>
> (magit :fetcher github
>        :repo "magit/magit"
>        :files ("lisp/magit"
>                "lisp/magit*.el"
>                "lisp/git-rebase.el"
>                "Documentation/magit.texi"
>                "Documentation/AUTHORS.md"
>                "LICENSE"
>          (:exclude "lisp/magit-libgit.el"
>        ;; Cannot remove this yet because it would
>        ;; also be removed from the stable version.
>        ;; "lisp/magit-section.el"
>        )))
>
> Repo magit/magit:
> https://github.com/magit/magit
>
> I have given you references, maybe I cannot read that well, so you can
> give me references to show if authors have participation in decision.
>

The available recipe options are all clearly documented in the README
for the melpa repository. Most maintainers don't use the :commit option
because it is extremely inconvenient, but it is there if they want it.
It is inconvenient because it means the recipe has to be updated, which
means a pull request has to be accepted before the package can be released.

Most maintainers will maintain a specific branch for releases. This is
normal practice in version control. Often, this is the master branch,
but 'release' and 'melpa' are also commonly used. Code is not pushed
onto these branches until it is ready for release. The package
maintainer has full control of this branch and therefore has full
control over when new code is released. This is also the model used by
GNU ELPA for external packages.

This is not the model you imply, where MELPA just grabs the data
whenever it wants and releases new version without management by the
package maintainer, resulting in the release of code that is not ready
for release.

--
Tim Cross

Re: Security issues in Emacs packages

[Greg Minshall]

Tim,

> I think you missed my point. There is no benefit in MELPA adopting
> signed packages because there is no formal code review and no vetting
> of the individuals who submit the code.

it occurs to me there might be one benefit: if George, whom you trust,
says, "I've been running version 1.2.3 of package xYandZ from MELPA and
i have a lot of confidence in it", then if you find that version of that
package with a trusted MELPA signature, you maybe know that you and
George are running the same software.  i.e., it helps with the "web of
trust" (if people still talk of that).

(so, the requirement for this is not audited packages, but a solid,
"secure", release procedure by MELPA.)

cheers, Greg

Re: Security issues in Emacs packages

[Jean Louis]

* Greg Minshall <minshall@umich.edu> [2020-11-26 08:29]:
> Tim,
> 
> > I think you missed my point. There is no benefit in MELPA adopting
> > signed packages because there is no formal code review and no vetting
> > of the individuals who submit the code.
> 
> it occurs to me there might be one benefit: if George, whom you trust,
> says, "I've been running version 1.2.3 of package xYandZ from MELPA and
> i have a lot of confidence in it", then if you find that version of that
> package with a trusted MELPA signature, you maybe know that you and
> George are running the same software.  i.e., it helps with the "web of
> trust" (if people still talk of that).
> 
> (so, the requirement for this is not audited packages, but a solid,
> "secure", release procedure by MELPA.)

Maybe principles from Freenet Web of Trust could be somehow
implemented for Emacs users and our discussions.
https://www.draketo.de/english/freenet/friendly-communication-with-anonymity

Re: Security issues in Emacs packages

[Tim Cross]

Greg Minshall <minshall@umich.edu> writes:

> Tim,
>
>> I think you missed my point. There is no benefit in MELPA adopting
>> signed packages because there is no formal code review and no vetting
>> of the individuals who submit the code.
>
> it occurs to me there might be one benefit: if George, whom you trust,
> says, "I've been running version 1.2.3 of package xYandZ from MELPA and
> i have a lot of confidence in it", then if you find that version of that
> package with a trusted MELPA signature, you maybe know that you and
> George are running the same software.  i.e., it helps with the "web of
> trust" (if people still talk of that).
>
> (so, the requirement for this is not audited packages, but a solid,
> "secure", release procedure by MELPA.)
>

It could, but to get that level of assurance, you not only have to
verify the signature is valid (something which is automated if enabled),
you also need to verify that both packages have the exact same
signature, which is pretty much a manual process. So in addition to
telling you the version number, George would also need to communicate
the signature and that would need to be compared to the signature you
have in the package you downloaded to know that the packages are in fact
the same (you cannot rely on version numbers for any real verification).

Signatures are a good thing and MELPA should implement them. However,
what they are really useful for is ensuring the package you have
downloaded has not been modified since it was created and signed.

--
Tim Cross

Re: Security issues in Emacs packages

[Greg Minshall]

Tim,

> It could, but to get that level of assurance, you not only have to
> verify the signature is valid (something which is automated if
> enabled), you also need to verify that both packages have the exact
> same signature, which is pretty much a manual process. So in addition
> to telling you the version number, George would also need to
> communicate the signature and that would need to be compared to the
> signature you have in the package you downloaded to know that the
> packages are in fact the same (you cannot rely on version numbers for
> any real verification).

if MELPA's release procedure prevented two separate releases of version
1.2.3 of package xYandZ from being released, wouldn't that obviate the
requirement for George to give me signatures?  that was my thought as to
why a signed (MELPA, version number, package name) would be enough.
(i've no idea if MELPA's procedures would actually conform to my
"requirement".)

cheers, Greg

Re: Security issues in Emacs packages

[Tim Cross]

Greg Minshall <minshall@umich.edu> writes:

> Tim,
>
>> It could, but to get that level of assurance, you not only have to
>> verify the signature is valid (something which is automated if
>> enabled), you also need to verify that both packages have the exact
>> same signature, which is pretty much a manual process. So in addition
>> to telling you the version number, George would also need to
>> communicate the signature and that would need to be compared to the
>> signature you have in the package you downloaded to know that the
>> packages are in fact the same (you cannot rely on version numbers for
>> any real verification).
>
> if MELPA's release procedure prevented two separate releases of version
> 1.2.3 of package xYandZ from being released, wouldn't that obviate the
> requirement for George to give me signatures?  that was my thought as to
> why a signed (MELPA, version number, package name) would be enough.
> (i've no idea if MELPA's procedures would actually conform to my
> "requirement".)
>

Possibly, but I'm not sure it does/can. From my limited understanding,
the version number is determined by the git tag (for stable packages - I
think the date is used for unstable). This is as it should be as it
should be the package maintainer who controls the version number, not
the packaging service (especially for maintainers who use semantic
versioning where the version number actually conveys information about
the package).

At the end of the day, this is essentially a supply chain problem. To
really have confidence, you need confidence in the whole supply chain,
not just the distribution centre.

Personally, I wish both GNU and Melpa had adopted a push mechanism for
package release. Something similar to npmjs.com where the package
author/maintainer would submit a signed package (publish) to the
repository. This would make it producers of the package code we trust,
not the distribution center (repository). Main downside with that
approach is you would also need a reliable mechanism for retrieving the
public keys (there would be a lot more of them to manage). I also think
this would be a model that is a lot easier to scale (something I think
GNU will have problems with under their current model.

--
Tim Cross

Re: Security issues in Emacs packages

[Jean Louis]

* Tim Cross <theophilusx@gmail.com> [2020-11-27 01:21]:
> 
> Greg Minshall <minshall@umich.edu> writes:
> 
> > Tim,
> >
> >> It could, but to get that level of assurance, you not only have to
> >> verify the signature is valid (something which is automated if
> >> enabled), you also need to verify that both packages have the exact
> >> same signature, which is pretty much a manual process. So in addition
> >> to telling you the version number, George would also need to
> >> communicate the signature and that would need to be compared to the
> >> signature you have in the package you downloaded to know that the
> >> packages are in fact the same (you cannot rely on version numbers for
> >> any real verification).
> >
> > if MELPA's release procedure prevented two separate releases of version
> > 1.2.3 of package xYandZ from being released, wouldn't that obviate the
> > requirement for George to give me signatures?  that was my thought as to
> > why a signed (MELPA, version number, package name) would be enough.
> > (i've no idea if MELPA's procedures would actually conform to my
> > "requirement".)
> >
> 
> Possibly, but I'm not sure it does/can. From my limited understanding,
> the version number is determined by the git tag (for stable packages - I
> think the date is used for unstable). This is as it should be as it
> should be the package maintainer who controls the version number, not
> the packaging service (especially for maintainers who use semantic
> versioning where the version number actually conveys information about
> the package).

Before some days or weeks we discussed this in a different thread, not
this mailing list. I think emacs-devel. Authors are by convention
writing their version numbers in their packages aligned to the Emacs
Lisp manual section on Packaging. MELPA is injecting their version
they are taking from git as commit number.

Thus MELPA does not use author's version number. It should be obvious
from package-list-packages that same version of the package in GNU
ELPA does not have same version number in MELPA, that is confusion
created for no good reason but to minimize programming efforts at
MELPA. 

> At the end of the day, this is essentially a supply chain problem. To
> really have confidence, you need confidence in the whole supply chain,
> not just the distribution centre.

Either way it could be good and depends what does the distribution
center do. If they are auditing packages and making sure of security
or are they just packaging without any audit. Maybe distribution
center verifies all PGP signatures and we may trust such center, maybe
not.

The OpenBSD software audits packages. It cannot ever be fully secure
but for base system one can rest assured that developers have put a
lot of effort in making it secure. Trust with users is then created
based on the relation and background of the OS distribution.

> Personally, I wish both GNU and Melpa had adopted a push mechanism for
> package release. Something similar to npmjs.com where the package
> author/maintainer would submit a signed package (publish) to the
> repository. This would make it producers of the package code we trust,
> not the distribution center (repository).

I wish that too.

Re: Security issues in Emacs packages

[Greg Minshall]

Tim,

> At the end of the day, this is essentially a supply chain problem. To
> really have confidence, you need confidence in the whole supply chain,
> not just the distribution centre.

that makes sense.  thanks.

Greg

Re: One vs many directories

[Jean Louis]

* Tom Gillespie <tgbugs@gmail.com> [2020-11-24 23:11]:
> > > That is security issue.
> >
> > Why is it a security issue? The variables do need to be close to the end
> > — 3000 characters is only about 50 lines.
> 
> It isn't a security issue by itself. Emacs never automatically runs
> eval file local variables unless you have tampered with
> enable-local-eval, in which case the tamperin is the security issue
> not the existence of the local variables list.
> 
> Thus it is only a security issue if you permanently accept that eval
> file local variable and then open random org files that use it with a
> malicious startup block. An eval file local variable like that which
> blindly executes an org babel block should never be permanently
> accepted

I do understand conditions.

But I can say that I did not understand conditions for one decade and
a half, as I was not aware that Emacs has a "real programming language
" built-in, and I have been spending my time with outside languages
that I was invoking from Emacs.

Yes, I did read that Emacs has Emacs Lisp. I was configuring Emacs but
I have not been thinkin that it is Lisp. I could figure out those
settings without reading manual.

As I am programming in Emacs Lisp for years I am aware of it. Before I
was thinking that local variables belong somewhere and that I should
enable it, despite all the warnings. There was lack of understanding
despite the information in front of me.

Some files opened asked me to enable local variables, so many times I
did so without thinking. My personal behavior to enable local
variables that other authors have written is probable not isolated
case. So that is security issue as number of users among thousands are
weak on this.

When I say security issue I do not think myself, you or majority of
people currently, but that there are probably millions of people who
can be affected by this. I also know spammers are harvesting mailing
lists.

org-sbe to automate some source block executions

[Jean Louis]

* Eric S Fraga <e.fraga@ucl.ac.uk> [2020-11-24 12:46]:
> On Tuesday, 24 Nov 2020 at 12:00, Jean Louis wrote:
> > Can I automated the execution of Babel code upon opening of the Org
> > file?
> 
> You can, by using file local variables.  For instance, for some files, I
> do this:
> 
> #+begin_src org
>   ,* local variables                         :noexport:
>   # Local Variables:
>   # eval: (org-sbe "startup")
>   # End:
> #+end_src
> 
> which will evaluate the named src block "startup" when file is
> opened.

I am trying to implement it, do you see here anything below that I am
doing wrong? Error is that it cannot find source block stages

** Stages
   
#+BEGIN_SRC sql :var results=stages :engine postgresql :exports results
SELECT -- salesflowstages_priority AS "Priority", 
       salesflowstages_publicstage AS "Development Stages" --,
--       salesflowstages_description AS "Description" 
FROM salesflowstages
WHERE salesflowstages_salesflows = 1 
ORDER BY salesflowstages_priority ASC
#+END_SRC

# Local Variables:
# eval: (org-sbe "stages")
# End:

Re: org-sbe to automate some source block executions

[Ihor Radchenko]

> I am trying to implement it, do you see here anything below that I am
> doing wrong? Error is that it cannot find source block stages

You should assign a name to your source block. Also, I am clueless why
you need a variable ":var results=stages". You should probably do
something like the following:

** Stages

#+name: stages   
#+BEGIN_SRC sql :engine postgresql :exports results
SELECT -- salesflowstages_priority AS "Priority", 
       salesflowstages_publicstage AS "Development Stages" --,
--       salesflowstages_description AS "Description" 
FROM salesflowstages
WHERE salesflowstages_salesflows = 1 
ORDER BY salesflowstages_priority ASC
#+END_SRC

# Local Variables:
# eval: (org-sbe "stages")
# End:

Best,
Ihor


Jean Louis <bugs@gnu.support> writes:

> * Eric S Fraga <e.fraga@ucl.ac.uk> [2020-11-24 12:46]:
>> On Tuesday, 24 Nov 2020 at 12:00, Jean Louis wrote:
>> > Can I automated the execution of Babel code upon opening of the Org
>> > file?
>> 
>> You can, by using file local variables.  For instance, for some files, I
>> do this:
>> 
>> #+begin_src org
>>   ,* local variables                         :noexport:
>>   # Local Variables:
>>   # eval: (org-sbe "startup")
>>   # End:
>> #+end_src
>> 
>> which will evaluate the named src block "startup" when file is
>> opened.
>
> I am trying to implement it, do you see here anything below that I am
> doing wrong? Error is that it cannot find source block stages
>
> ** Stages
>    
> #+BEGIN_SRC sql :var results=stages :engine postgresql :exports results
> SELECT -- salesflowstages_priority AS "Priority", 
>        salesflowstages_publicstage AS "Development Stages" --,
> --       salesflowstages_description AS "Description" 
> FROM salesflowstages
> WHERE salesflowstages_salesflows = 1 
> ORDER BY salesflowstages_priority ASC
> #+END_SRC
>
> # Local Variables:
> # eval: (org-sbe "stages")
> # End:

Re: org-sbe to automate some source block executions

[Jean Louis]

* Ihor Radchenko <yantar92@gmail.com> [2020-11-25 14:40]:
> > I am trying to implement it, do you see here anything below that I am
> > doing wrong? Error is that it cannot find source block stages
> 
> You should assign a name to your source block. Also, I am clueless why
> you need a variable ":var results=stages". You should probably do
> something like the following:

Because I did not know #+NAME: is there for source block names. Now I
have solved it all.

As I am transitioning with tasks it becomes also little questionable
how to structure the rest of files.

Re: One vs many directories

[Jean Louis]

* Eric S Fraga <e.fraga@ucl.ac.uk> [2020-11-24 12:46]:
> On Tuesday, 24 Nov 2020 at 12:00, Jean Louis wrote:
> > Can I automated the execution of Babel code upon opening of the Org
> > file?
> 
> You can, by using file local variables.  For instance, for some files, I
> do this:
> 
> #+begin_src org
>   ,* local variables                         :noexport:
>   # Local Variables:
>   # eval: (org-sbe "startup")
>   # End:
> #+end_src

I have got it to work as I had to name the source block. It does
evaluates and I get the result in the message buffer, but it does not
expands in the Org buffer. That is what I wish to find out how.

** Stages
#+NAME: stages   
#+BEGIN_SRC sql :engine postgresql :exports results :results replace
SELECT 1 AS table;
#+END_SRC

# Local Variables:
# eval: (org-sbe "stages")
# End:

Re: One vs many directories

[Eric S Fraga]

On Wednesday, 25 Nov 2020 at 14:46, Jean Louis wrote:
> I have got it to work as I had to name the source block. 

Yes, org-sbe looks for the first source block with the name
given.  Nothing to do with headings etc.

> It does evaluates and I get the result in the message buffer, but it
> does not expands in the Org buffer. That is what I wish to find out
> how.

What do you get on the same src block if you explicitly C-c C-c there?

-- 
: Eric S Fraga via Emacs 28.0.50, Org release_9.4-118-g2a4578.dirty

Re: One vs many directories

[Jean Louis]

* Eric S Fraga <e.fraga@ucl.ac.uk> [2020-11-25 16:08]:
> On Wednesday, 25 Nov 2020 at 14:46, Jean Louis wrote:
> > I have got it to work as I had to name the source block. 
> 
> Yes, org-sbe looks for the first source block with the name
> given.  Nothing to do with headings etc.
> 
> > It does evaluates and I get the result in the message buffer, but it
> > does not expands in the Org buffer. That is what I wish to find out
> > how.
> 
> What do you get on the same src block if you explicitly C-c C-c there?

I get results in the buffer. That is what I would like, to get
expanded results in the buffer upon opening of the Org file.

Re: One vs many directories

[Ihor Radchenko]

> ... It does
> evaluates and I get the result in the message buffer, but it does not
> expands in the Org buffer.

It is expected behaviour. According to the docstring of org-sbe, it only
returns the value, but does not actually change buffer.

If you want to replace the RESULTS, you need to use the following:

# Local Variables:
# eval: (let ((pos (org-babel-find-named-block "stages"))) (when pos (save-excursion (org-with-point-at pos (org-babel-execute-src-block)))))
# End:

Best,
Ihor


Jean Louis <bugs@gnu.support> writes:

> * Eric S Fraga <e.fraga@ucl.ac.uk> [2020-11-24 12:46]:
>> On Tuesday, 24 Nov 2020 at 12:00, Jean Louis wrote:
>> > Can I automated the execution of Babel code upon opening of the Org
>> > file?
>> 
>> You can, by using file local variables.  For instance, for some files, I
>> do this:
>> 
>> #+begin_src org
>>   ,* local variables                         :noexport:
>>   # Local Variables:
>>   # eval: (org-sbe "startup")
>>   # End:
>> #+end_src
>
> I have got it to work as I had to name the source block. It does
> evaluates and I get the result in the message buffer, but it does not
> expands in the Org buffer. That is what I wish to find out how.
>
> ** Stages
> #+NAME: stages   
> #+BEGIN_SRC sql :engine postgresql :exports results :results replace
> SELECT 1 AS table;
> #+END_SRC
>
> # Local Variables:
> # eval: (org-sbe "stages")
> # End:

Re: One vs many directories

[Jean Louis]

* Ihor Radchenko <yantar92@gmail.com> [2020-11-25 16:16]:
> 
> > ... It does
> > evaluates and I get the result in the message buffer, but it does not
> > expands in the Org buffer.
> 
> It is expected behaviour. According to the docstring of org-sbe, it only
> returns the value, but does not actually change buffer.
> 
> If you want to replace the RESULTS, you need to use the following:
> 
> # Local Variables:
> # eval: (let ((pos (org-babel-find-named-block "stages"))) (when pos (save-excursion (org-with-point-at pos (org-babel-execute-src-block)))))
> # End:

That works well, thank you for the tip.

Of course I will not be writing all that by hand, program would simply
invoke the creation and generation of Org file and write headings and
the local variables. Next time I open the Org file related to that
person I can see all the pending tasks or tasks done with hyperlinks
to their corresponding actual tracking places in the database.

I have already made the function to capture Org tasks into the
database, concept is here:

(defun hyperscope-capture-org-heading ()
  "Captures Org heading and stores it in the Hyperscope dynamic
  knowledge repository"
  (interactive)
  (let* ((body (org-copy-subtree nil nil nil t))
	 (body (pop kill-ring))
	 (body (org-no-properties body))
	 (heading (org-get-heading))
	 (created (org-property-values "CREATED"))
	 (date (if created (substring (car created) 1 11) nil))
	 (body (with-temp-buffer
		 (insert body)
		 (org-mode)
		 (org-back-to-heading)
		 (kill-line)
		 (delete-matching-lines ":PROPERTIES:")
		 (delete-matching-lines ":CREATED:")
		 (delete-matching-lines ":ID:")
		 (delete-matching-lines ":END:")
		 (buffer-string))))
    (hyperscope-add-note-to-set heading body date)))

So I am now transitioning all Org tasks into little bit different and
more streamlined system for me personally.

Nevertheless, when I use a browser I can still use org-capture and
later just parse all headings automatically and insert into the
database.

Re: One vs many directories

[Dr. Arne Babenhauserheide]

[-- Attachment #1: Type: text/plain, Size: 2026 bytes --]


Jean Louis <bugs@gnu.support> writes:

> Some people maybe access multiple Org files through Agenda, me I
> don't. Some items are "non existent" and I do not know how to ask
> agenda to refresh itself.

Simply press the letter g.

For my own setup I run code in a hook to update the agenda whenever I
change a TODO state, clock in or clock out, but that has performance
problems when I do it while the Agenda is shown.

    (defvar todo-modified-from-agenda nil "Indicates whether org-mode todo state changes were triggered from the agenda. Check this to avoid trying to propagate the change back into the agenda")
    ;; continuously update agenda view, from http://thomasf.github.io/solarized-css/test/org-hacks.html
    (defun kiwon/org-agenda-redo-in-other-window ()
      "Call org-agenda-redo function even in the non-agenda buffer."
      (interactive)
      (when (not (and (boundp 'todo-modified-from-agenda) todo-modified-from-agenda)) ; org-state non-nil means we’re coming from the org-after-todo-state-change-hook, which would throw when changing todo states from agenda due to a circular action
        (let ((agenda-window (get-buffer-window (or (and (boundp 'org-agenda-buffer-name) org-agenda-buffer-name) "plan.org") t)))
          (when agenda-window
    	(with-selected-window agenda-window
    	  (org-agenda-redo))))))
    ;; advice agenda todo to avoid redo, thanks to http://nullprogram.com/blog/2013/01/22/
    (defadvice org-agenda-todo (before org-agenda-disable-redo activate)
      (setq todo-modified-from-agenda t))
    (defadvice org-agenda-todo (after org-agenda-enable-redo activate)
      (setq todo-modified-from-agenda nil))
    
    (add-hook 'org-clock-in-hook 'kiwon/org-agenda-redo-in-other-window)
    (add-hook 'org-clock-out-hook 'kiwon/org-agenda-redo-in-other-window)
    (add-hook 'org-after-todo-state-change-hook 'kiwon/org-agenda-redo-in-other-window)


Best wishes,
Arne
-- 
Unpolitisch sein
heißt politisch sein
ohne es zu merken

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 1125 bytes --]

Re: One vs many directories

[Jean Louis]

* Dr. Arne Babenhauserheide <arne_bab@web.de> [2020-11-24 21:48]:
> 
> Jean Louis <bugs@gnu.support> writes:
> 
> > Some people maybe access multiple Org files through Agenda, me I
> > don't. Some items are "non existent" and I do not know how to ask
> > agenda to refresh itself.
> 
> Simply press the letter g.

What function is on g on your side?

I press g and I get error: invalid key g

> For my own setup I run code in a hook to update the agenda whenever I
> change a TODO state, clock in or clock out, but that has performance
> problems when I do it while the Agenda is shown.

That sounds that it will use computing power. Thank you. I have plan
to switch anything action related to database system and use Org to
view tasks, not to handle or store them.

Re: One vs many directories

[Dr. Arne Babenhauserheide]

[-- Attachment #1: Type: text/plain, Size: 1151 bytes --]


Jean Louis <bugs@gnu.support> writes:

> * Dr. Arne Babenhauserheide <arne_bab@web.de> [2020-11-24 21:48]:
>> 
>> Jean Louis <bugs@gnu.support> writes:
>> 
>> > Some people maybe access multiple Org files through Agenda, me I
>> > don't. Some items are "non existent" and I do not know how to ask
>> > agenda to refresh itself.
>> 
>> Simply press the letter g.
>
> What function is on g on your side?

(org-agenda-redo-all &optional EXHAUSTIVE)

> I press g and I get error: invalid key g
>
>> For my own setup I run code in a hook to update the agenda whenever I
>> change a TODO state, clock in or clock out, but that has performance
>> problems when I do it while the Agenda is shown.
>
> That sounds that it will use computing power. 

Yes, it does, but it gives me the interaction I want.

> Thank you. I have plan
> to switch anything action related to database system and use Org to
> view tasks, not to handle or store them.

That might cost you reaction-time in the end, because org then cannot
just keep the file open.

Best wishes,
Arne
-- 
Unpolitisch sein
heißt politisch sein
ohne es zu merken

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 1125 bytes --]

Re: One vs many directories

[Jean Louis]

* Dr. Arne Babenhauserheide <arne_bab@web.de> [2020-11-25 11:14]:
> 
> Jean Louis <bugs@gnu.support> writes:
> 
> > * Dr. Arne Babenhauserheide <arne_bab@web.de> [2020-11-24 21:48]:
> >> 
> >> Jean Louis <bugs@gnu.support> writes:
> >> 
> >> > Some people maybe access multiple Org files through Agenda, me I
> >> > don't. Some items are "non existent" and I do not know how to ask
> >> > agenda to refresh itself.
> >> 
> >> Simply press the letter g.
> >
> > What function is on g on your side?
> 
> (org-agenda-redo-all &optional EXHAUSTIVE)

When I do C-c a it runs (org-agenda) but I do not have "g" and I am on
development version. The C-c a window is made so that I cannot go with
cursor inside and that I cannot even expect the key map neither invoke
command by M-x and I cannot even M-:

All that is wrong and not aligned to Emacs common interface. It is bug
that bugs. Agenda buffer should allow users those standard Emacs
features.

> > Thank you. I have plan to switch anything action related to
> > database system and use Org to view tasks, not to handle or store
> > them.
> 
> That might cost you reaction-time in the end, because org then cannot
> just keep the file open.

Do you mean my reaction time or Org reaction time? I did not
understand about to keep file open, how do you mean?

You remember I said Org files with tasks are on my side almost always
related to some people, among them I am included. So I find Org file
per person in the directory wherever it is by clicking F4. Then task
list in the Org file would update itself as I have got the tip on how
to execute the SQL block. Do you mean that SQL block would cost me
reaction time? If so, that can be. In the Hyperscope database managed
by Emacs and displayed within tabulated-list-mode on this T410
Thinkpad the list of entries looks almost instant, I just press left
or right and I get new list. Each time there is SQL query.

Without you telling me I would not be thinking on that, but it is
true, the larger or more complex the query then it could be
processing.

But if there is large list of tasks not done, that means anyway that
something is terribly wrong with it.

Those tasks done, can appear under different heading. But then I
cannot selectively automatically execute block under one heading at
file opening time while not executing block under different heading.

Re: One vs many directories

[Ihor Radchenko]

> When I do C-c a it runs (org-agenda) but I do not have "g" and I am on
> development version. The C-c a window is made so that I cannot go with
> cursor inside and that I cannot even expect the key map neither invoke
> command by M-x and I cannot even M-:

C-c a will first show so-called agenda dispatcher asking you what kind
of agenda view you want to get. You need to press a key according to
the popup window (i.e. `t' to see all not done items). Then, you will
get the proper agenda buffer with all the keymaps set and `g' bound to
refreshing the chosen agenda view in the buffer.

> All that is wrong and not aligned to Emacs common interface. It is bug
> that bugs. Agenda buffer should allow users those standard Emacs
> features.

I am wondering what is the common Emacs interface you refer to. I am not
aware about any standard way to prompt user while also showing detailed
description of what to expect from different choices.

Best,
Ihor

Jean Louis <bugs@gnu.support> writes:

> * Dr. Arne Babenhauserheide <arne_bab@web.de> [2020-11-25 11:14]:
>> 
>> Jean Louis <bugs@gnu.support> writes:
>> 
>> > * Dr. Arne Babenhauserheide <arne_bab@web.de> [2020-11-24 21:48]:
>> >> 
>> >> Jean Louis <bugs@gnu.support> writes:
>> >> 
>> >> > Some people maybe access multiple Org files through Agenda, me I
>> >> > don't. Some items are "non existent" and I do not know how to ask
>> >> > agenda to refresh itself.
>> >> 
>> >> Simply press the letter g.
>> >
>> > What function is on g on your side?
>> 
>> (org-agenda-redo-all &optional EXHAUSTIVE)
>
> When I do C-c a it runs (org-agenda) but I do not have "g" and I am on
> development version. The C-c a window is made so that I cannot go with
> cursor inside and that I cannot even expect the key map neither invoke
> command by M-x and I cannot even M-:
>
> All that is wrong and not aligned to Emacs common interface. It is bug
> that bugs. Agenda buffer should allow users those standard Emacs
> features.
>
>> > Thank you. I have plan to switch anything action related to
>> > database system and use Org to view tasks, not to handle or store
>> > them.

Re: One vs many directories

[Jean Louis]

* Ihor Radchenko <yantar92@gmail.com> [2020-11-25 14:48]:
> > When I do C-c a it runs (org-agenda) but I do not have "g" and I am on
> > development version. The C-c a window is made so that I cannot go with
> > cursor inside and that I cannot even expect the key map neither invoke
> > command by M-x and I cannot even M-:
> 
> C-c a will first show so-called agenda dispatcher asking you what kind
> of agenda view you want to get. You need to press a key according to
> the popup window (i.e. `t' to see all not done items). Then, you will
> get the proper agenda buffer with all the keymaps set and `g' bound to
> refreshing the chosen agenda view in the buffer.
> 
> > All that is wrong and not aligned to Emacs common interface. It is bug
> > that bugs. Agenda buffer should allow users those standard Emacs
> > features.
> 
> I am wondering what is the common Emacs interface you refer to. I am not
> aware about any standard way to prompt user while also showing detailed
> description of what to expect from different choices.

Sorry for that vague expression. Let us say I open Completions buffer
I can switch into it, inspect it, ask for defined keys, evaluate with
M-:, Emacs allows me to remain in the window and go to other
window. Agenda buffer does not do that, this is probably because it
just waits for any key and does not allow anything else. That means I
will open Agenda and I cannot switch to other window, so I will close
agenda to switch. Maybe I have 10 TODO keywords, I have to open file,
I open Agenda again, aha, T... then close agenda, open file see
keyword, then open agenda again. Repetitions without end and user is
unable to use multiple windows. This really need not be so.

If I open packages' list I can keep the buffer and move to other
buffer while looking into that list.

You know how agenda is made like the 1985 BASIC menu in Commodore
C=64, but even back then there was better interface for such menus.

Re: One vs many directories

[Ihor Radchenko]

> Sorry for that vague expression. Let us say I open Completions buffer
> I can switch into it, inspect it, ask for defined keys, evaluate with
> M-:, Emacs allows me to remain in the window and go to other
> window. Agenda buffer does not do that, this is probably because it
> just waits for any key and does not allow anything else. That means I
> will open Agenda and I cannot switch to other window, so I will close
> agenda to switch. Maybe I have 10 TODO keywords, I have to open file,
> I open Agenda again, aha, T... then close agenda, open file see
> keyword, then open agenda again. Repetitions without end and user is
> unable to use multiple windows. This really need not be so.

It appears to me (correct me if I am wrong) that you haven't tried
agenda multi-occur/keyword/etc searches. The way it works is that you
only need to select type of search from agenda dispatcher window (the
one you are criticising). Later, when you actually enter the search
string, you are left with an ordinary Emacs prompt. You are free to
leave the minibuffer and switch to other buffers searching for the
keywords you are interested in.

In general, there are two types of agenda searches available from agenda
dispatcher:
1. Most are free-hand searches where you need to type a search string:
   - match TAGS/PROP/TODO query
   - search for keywords
   - multi-occur
   - first two searches, but limited only to TODO tasks
2. Pre-defined searches where search string was set in advance (by
   developers or by user via org-agenda-custom-commands):
   - agenda listing tasks relevant to current week or day
   - all TODO entries
   - FLAGGED entries
   - stuck projects

The first type will let you leave search prompt as soon as you select
what type of search you are about to enter.
The second type does not really need entering any extra text - it is
predefined.

> You know how agenda is made like the 1985 BASIC menu in Commodore
> C=64, but even back then there was better interface for such menus.

FYI. Transient.el - menu dispatcher in popular magit package also does
not let you switch buffers. So, apparently many people would not agree
that it is so terrible design.

P.S. Nothing prevents you from calling, i.e. M-x org-occur or binding it
to a key of your choice.

Best,
Ihor

Jean Louis <bugs@gnu.support> writes:

> * Ihor Radchenko <yantar92@gmail.com> [2020-11-25 14:48]:
>> > When I do C-c a it runs (org-agenda) but I do not have "g" and I am on
>> > development version. The C-c a window is made so that I cannot go with
>> > cursor inside and that I cannot even expect the key map neither invoke
>> > command by M-x and I cannot even M-:
>> 
>> C-c a will first show so-called agenda dispatcher asking you what kind
>> of agenda view you want to get. You need to press a key according to
>> the popup window (i.e. `t' to see all not done items). Then, you will
>> get the proper agenda buffer with all the keymaps set and `g' bound to
>> refreshing the chosen agenda view in the buffer.
>> 
>> > All that is wrong and not aligned to Emacs common interface. It is bug
>> > that bugs. Agenda buffer should allow users those standard Emacs
>> > features.
>> 
>> I am wondering what is the common Emacs interface you refer to. I am not
>> aware about any standard way to prompt user while also showing detailed
>> description of what to expect from different choices.
>
> Sorry for that vague expression. Let us say I open Completions buffer
> I can switch into it, inspect it, ask for defined keys, evaluate with
> M-:, Emacs allows me to remain in the window and go to other
> window. Agenda buffer does not do that, this is probably because it
> just waits for any key and does not allow anything else. That means I
> will open Agenda and I cannot switch to other window, so I will close
> agenda to switch. Maybe I have 10 TODO keywords, I have to open file,
> I open Agenda again, aha, T... then close agenda, open file see
> keyword, then open agenda again. Repetitions without end and user is
> unable to use multiple windows. This really need not be so.
>
> If I open packages' list I can keep the buffer and move to other
> buffer while looking into that list.
>
> You know how agenda is made like the 1985 BASIC menu in Commodore
> C=64, but even back then there was better interface for such menus.

Re: One vs many directories

[Jean Louis]

* Ihor Radchenko <yantar92@gmail.com> [2020-11-26 16:29]:
> > Sorry for that vague expression. Let us say I open Completions buffer
> > I can switch into it, inspect it, ask for defined keys, evaluate with
> > M-:, Emacs allows me to remain in the window and go to other
> > window. Agenda buffer does not do that, this is probably because it
> > just waits for any key and does not allow anything else. That means I
> > will open Agenda and I cannot switch to other window, so I will close
> > agenda to switch. Maybe I have 10 TODO keywords, I have to open file,
> > I open Agenda again, aha, T... then close agenda, open file see
> > keyword, then open agenda again. Repetitions without end and user is
> > unable to use multiple windows. This really need not be so.
> 
> It appears to me (correct me if I am wrong) that you haven't tried
> agenda multi-occur/keyword/etc searches.

I know multi-occur and tried it. Functions of org-agenda are useful
while initial menu window of org-agenda is so much less useful and I
speak of menu, not of individual functions. If it is Org agenda, I see
no reason why it could not be displayed in Org mode being read-only
and having buffer not killed when user press q.

We spoke of standard Emacs way. Normally buffers are not killed. If I
press q in Dired buffer is just buried, not killed. I find Org agenda
useful and I would be maybe moving to *Agenda Commands* buffer by
using (next-buffer) and (prev-buffer) and those are key bindings like
hyper key - move-end-of-line and hyper key - move-beginning of line.

It is my standard behavior to move to one buffer and go to other
buffer.

User could have one tab open for Org agenda menu, other tab for
work. Why invoke all time C-c a to access org agenda.

I am not speaking for me personally, I am giving you clear comparison
of that menu to mu4e or org-agenda menu and other pop-up features that
do not block user in using Emacs. Org agenda menu blocks the
interface!

When invoking package-list-packages it does not block me moving to
other buffers and also offers complex menu system and key
bindings. Buffer is not killed but burried when I press q.

Functions of org agenda are fine, this is reference to blocking of
Emacs interface while opening Org agenda buffer. It diminishes
usability.

I guess nobody cares. It is how it is. No need to consult external
references such as
https://www.nngroup.com/articles/usability-101-introduction-to-usability/

No need even to look how other parts of Emacs are interacting with
user and to harmonize the menu. 

> > You know how agenda is made like the 1985 BASIC menu in Commodore
> > C=64, but even back then there was better interface for such menus.
> 
> FYI. Transient.el - menu dispatcher in popular magit package also does
> not let you switch buffers. So, apparently many people would not agree
> that it is so terrible design.

Compare things to things that are better not to things that are
worse. 

You have to read more about usability. To know what is the problem you
have to research and not assume that if nobody says anything that it
is usable enough. Nobody complains. I did not hear complain, so it
must be perfect. Irony. That is not how interfaces get improved and
definitely not how developer would discover if something is wrong.

If developer is waiting for somebody to complain that will be too
late. Thousands of users will have problems that were not told to
developer.

Was there any methodology to decide what is good user menu?

Reference:
https://www.nngroup.com/articles/why-you-only-need-to-test-with-5-users/

Was there any survey here to ask about any usability?!  Hypothetical
question.

> P.S. Nothing prevents you from calling, i.e. M-x org-occur or binding it
> to a key of your choice.

Personally I know that. I speak for general usability. You know that I
have my system of doing things with or without Org mode.

Re: One vs many directories

[Jean Louis]

* Ihor Radchenko <yantar92@gmail.com> [2020-11-25 14:48]:
> > When I do C-c a it runs (org-agenda) but I do not have "g" and I am on
> > development version. The C-c a window is made so that I cannot go with
> > cursor inside and that I cannot even expect the key map neither invoke
> > command by M-x and I cannot even M-:
> 
> C-c a will first show so-called agenda dispatcher asking you what kind
> of agenda view you want to get. You need to press a key according to
> the popup window (i.e. `t' to see all not done items). Then, you will
> get the proper agenda buffer with all the keymaps set and `g' bound to
> refreshing the chosen agenda view in the buffer.
> 
> > All that is wrong and not aligned to Emacs common interface. It is bug
> > that bugs. Agenda buffer should allow users those standard Emacs
> > features.
> 
> I am wondering what is the common Emacs interface you refer to. I am not
> aware about any standard way to prompt user while also showing detailed
> description of what to expect from different choices.

It is maybe not standard, but I never expected in last 20 years to get
blocked when having some window as menu in front of me.

Please look how mu4e is showing menu and compare:

1. when I open mu4e the menu does not block me to divide screen in 2
  windows

2. org-agenda blocks me, it denies me using Emacs interface. This is
   personally disturbing and makes accessing Org agenda repetitive

Observe how C-x C-f tries to find file:

1. it opens minibuffer and does not disturb user to move from
   minibuffer to other buffer to find references. I often have file
   names in other buffers and I move to minibuffer to open specific
   file

2. org-agenda does not allow any movement

It is usability question. Personally I do not mind as I am
transitioning and using Org files not anymore for planning, rather for
documents.

Org agenda window shall simply get a focus and be displayed in
read-only Org mode with few key bindings invoking those commands. This
way both the minibuffer and other windows remain accessible.

I have tried to be nice when describing my experience with it. In very
kind way I can say that I do not find it usable.

Reference:
https://www.nngroup.com/articles/usability-101-introduction-to-usability/

For me it was not easy for reason that org-agenda offers 16 different
menues and that I cannot keep open org-agenda window and move to other
window for references. It requires me to write notes on paper to be
able to use org-agenda. When searching for things I often use other
window, I do copy and paste into minibuffer, read info files or other
buffers.

Consider this a bug, and it is already here on the mailing list.

Jean

Re: One vs many directories

[Ihor Radchenko]

> For my own setup I run code in a hook to update the agenda whenever I
> change a TODO state, clock in or clock out, but that has performance
> problems when I do it while the Agenda is shown.

You do not have or update the whole agenda view.

I use the following code to update the clocking highlights in agenda
even when I clock-in/out outside the agenda buffer:

https://github.com/yantar92/emacs-config/blob/master/config.org#update-highlight-from-currently-clocked-task-in-agenda-even-if-the-task-was-clocked-inout-from-outside

The same can be done for todo state changes using
org-agenda-change-all-lines

Best,
Ihor

"Dr. Arne Babenhauserheide" <arne_bab@web.de> writes:

> Jean Louis <bugs@gnu.support> writes:
>
>> Some people maybe access multiple Org files through Agenda, me I
>> don't. Some items are "non existent" and I do not know how to ask
>> agenda to refresh itself.
>
> Simply press the letter g.
>
> For my own setup I run code in a hook to update the agenda whenever I
> change a TODO state, clock in or clock out, but that has performance
> problems when I do it while the Agenda is shown.
>
>     (defvar todo-modified-from-agenda nil "Indicates whether org-mode todo state changes were triggered from the agenda. Check this to avoid trying to propagate the change back into the agenda")
>     ;; continuously update agenda view, from http://thomasf.github.io/solarized-css/test/org-hacks.html
>     (defun kiwon/org-agenda-redo-in-other-window ()
>       "Call org-agenda-redo function even in the non-agenda buffer."
>       (interactive)
>       (when (not (and (boundp 'todo-modified-from-agenda) todo-modified-from-agenda)) ; org-state non-nil means we’re coming from the org-after-todo-state-change-hook, which would throw when changing todo states from agenda due to a circular action
>         (let ((agenda-window (get-buffer-window (or (and (boundp 'org-agenda-buffer-name) org-agenda-buffer-name) "plan.org") t)))
>           (when agenda-window
>     	(with-selected-window agenda-window
>     	  (org-agenda-redo))))))
>     ;; advice agenda todo to avoid redo, thanks to http://nullprogram.com/blog/2013/01/22/
>     (defadvice org-agenda-todo (before org-agenda-disable-redo activate)
>       (setq todo-modified-from-agenda t))
>     (defadvice org-agenda-todo (after org-agenda-enable-redo activate)
>       (setq todo-modified-from-agenda nil))
>     
>     (add-hook 'org-clock-in-hook 'kiwon/org-agenda-redo-in-other-window)
>     (add-hook 'org-clock-out-hook 'kiwon/org-agenda-redo-in-other-window)
>     (add-hook 'org-after-todo-state-change-hook 'kiwon/org-agenda-redo-in-other-window)
>
>
> Best wishes,
> Arne
> -- 
> Unpolitisch sein
> heißt politisch sein
> ohne es zu merken

Re: One vs many directories

[Ihor Radchenko]

> Can I automated the execution of Babel code upon opening of the Org
> file?

Adding to other suggestions, you can always add a custom function to
org-mode-hook instead of playing with file-local variables.

> Then we comes to actual execution of tasks. How do we get reminded?
>
> Is the reminder only if I press {C-c a} for org-agenda? Do I need to
> do action to get reminded?

You can always configure Emacs to run agenda on startup. Just add a
command to your init file ;)

For automatic reminders, there is built-in org-notify.el or external
org-alert package (https://github.com/spegoraro/org-alert).

> Personal problem is that tasks are sparse and separate in various Org
> files and not centralized. I become dependent of org-agenda to do what
> I need but it never does what I need.

I agree that org-agenda has many issues that cannot be easily solved
because of its complexity. However, everything you describe (including
multi-occur) can also be achieved with org-ql
(https://github.com/alphapapa/org-ql) - analogue of SQL query language
for org-mode (with more optimisations in comparison with org-agenda). 

Best,
Ihor

Jean Louis <bugs@gnu.support> writes:

> * Ihor Radchenko <yantar92@gmail.com> [2020-11-23 08:43]:
>> >> I am wondering what you mean by Org's philosophy. Why would it have anything to do with directories?
>> >
>> > Org's philosophy is to have one or a handful of directories without
>> > nesting of directories.  Users are not expected to have their Org
>> > files in a deeply nested tree.  Org also prefers big files with large
>> > trees rather than lots of little files.
>> >
>> > By philosophy, I mean the dev consensus on the correct way to do
>> > things, and coded configuration and usability biases.
>> 
>> I believe that org support all possibilities. The user can decide to
>> keep many (possibly nested) org files, a few large org files, or
>> anywhere in between. There are several parallel feature sets allowing to
>> work in a single file as well as with a bunch of smaller files.
>
> Yes, sure, and I guess you mentioned some people have problems with
> many files. And I have no problem at all with many files as they are
> per subject separated, per person or per subject separated. They are
> not hyperlinked to each other, it is me who make system to hyperlink
> to files.
>
> Searching for Joe Doe, F4 and I am in Org file for Joe Doe. My
> personal TODO list need not really show the tasks assigned to Joe Doe,
> I could show only * TODO Joe Doe and when I click there then I can get
> all tasks for Joe Doe as new Org file.
>
> It means I am accessing hundreds of Org files from the meta level by
> using conceptual location backed by the database.
>
> Some people maybe access multiple Org files through Agenda, me I
> don't. Some items are "non existent" and I do not know how to ask
> agenda to refresh itself. This is not big deal as I do not access
> items throgh Agenda, though I find it very useful. 
>
> org-agenda is trying to put all tasks and notes from various files
> into one list and that is of course not so easy task considering that
> files can be anywhere on the file system and that they need to be
> "remembered". 
>
>> For a single file, the user can search headings with org-goto (without a
>> need to explicitly travel through all the nesting headline levels),
>> reveal only headings satisfying certain keyword/tag/any other search
>> criteria with org-sparse-tree, or built agenda views restricted to a
>> single file (or even subtree).
>
> M-x org-goto is useful feature to find headlines. And I never use it,
> just standard Emacs search is enough within a file. Meanings I am
> searching are often inside of the headline. And it is not my perosnal
> way locating things.
>
> Personally, I am using parts of Org, like specific headling to export
> it and to send to remote person, or to print the file as project and
> to bind it nicely.
>
> When I see repetitive action, for example that I have to send "Daily
> Report" template to a person by email, than I just bookmark that in
> Emacs with {C-x r m} under something that I think is the meaning of
> it. Then I forget about it. Next time when I need to send report, I am
> {C-x r b} and quickly completing it and then I am exporting and
> inserting into the email.
>
> In general I speak of subsets or sub-lists among lists. List of Org
> files is one list, list of headings within Org file is other list, and
> list of specific subject related headings or bookmarks to such is
> third subset of lists.
>
>> For multiple files located anywhere in the filesystem, there is always
>> org-refile capable of filing the information to proper place
>> searching deeply nested headlines with ease regardless of the file the
>> information is physically located in. Headlines from multiple files can
>> be grouped using agenda views for any given search criteria (showing
>> todo items or items for a single day/week is just a tiny subset of what
>> agenda can do).
>
> That may be useful for those who find it while my use case is
> different, here is how it is for me:
>
> ** TODO Heading [1/2] [50%]
>
> 1) [X] Do this
> 2) [ ] Do that
>
> that is my personal use case. I do not do things like:
>
> ** TODO Do this
>
> Description of the task
>
> And so far I know org-refile works on headings. It does not work on
> list items.
>
> Sometimes the task describes something that belongs to other file, I
> just kill and yank to other file. And I keep RCS revision control
> system of files.
>
> As user may have many various sparse tasks to do or notes that require
> action and attention in soonest future it is best to consolidate tasks
> into one centralized system.
>
> Such system should encompass all tasks or notes that require attention
> or action in soonest future and should offer constant reminders to
> user on what has to be done and when and which people are related to
> the task.
>
> When I mentioned "sparse tasks" I refer to my usage and handling of
> mess:
>
> 1) Bunch of Org files, org-agenda and Org mode tries to accommodate me
>    by consolidating everything into lists
>
> 2) There are hundreds of such tasks all over, Org tries to consolidate
>    it.
>
> 3) There are various tasks and actions to do that are not recorded in
>    Org files, those cannot be handled by Org.
>
> 4) There are database based Tasks in several groupings, some are just
>    tagged with TODO, some are recorded in actual action requiring
>    groups.
>
> Instead of attempting to be perfect by using Org files for me
> personally is best to consolidate all tasks in the database as such
> are related either to people mostly or to some subjects related to me
> personally which again belongs to "People".
>
> And Org file for people need not have a task there, it can be exported
> automatically into the Org.
>
> - when searching for Person Robert S., I locate person and press F4
>
> - Org file for that person is automatically created
>
> - heading * Tasks is automatically created and expanded there by using SQL
>
> Concept is here:
>
> * Tasks
>
> #+BEGIN_SRC sql :engine postgresql :exports results :results value raw 
> SELECT '** [[(todo ' || simpletodos_id || ')][TODO]] ' || simpletodos_datecreated::date || ' ' ||
> simpletodos_name || E'\n\n' || simpletodos_description FROM simpletodos
> WHERE simpletodos_contacts = 23187;
> #+END_SRC
>
> #+RESULTS:
> ** [[(todo120)][TODO]] 2017-11-07 Do something
>
> Something I need to do. (THIS IS HEADING OR TASK DESCRIPTION).
>
> The SQL query need to defined only one time and not in the Org file,
> but in the program that creates the Org file for the user. Instead of
> the SQL query in the specific Org file it could be a simple Emacs
> expression that never changes such as (tasks-for-user)
>
> Then the SQL query generates the headings under #+RESULTS: and those
> headings come from centralized consolidated database of tasks.
>
> It is then interesting that this column of the database can include
> any kind of the mode that Emacs supports, it could include any kind of
> file, be it image, video or anything as long as such task is assigned
> to the user. It could include other Org files and collection of Org
> files or just description or database based whole Org file if
> necessary. It becomes very abstract and liberated from limits.
>
> That way I would be editing tasks on the meta level by clicking on
> TODO link. If task would be done and completed it could be shown in
> separate section of Org file related to user.
>
> Additional buttons can be automatically included such as:
>
> - All tasks for user -> Hyperlink to meta level consolidated TODO management
>
> - Add task for user
>
> - Re-assign from this user to other user
>
> That is using Org mode as viewer of various information, not only as
> handler of the information.
>
> The cruft with org-agenda is then removed as then by pressing F5 I get
> the list of all tasks and I could filter it how I wish. Which is
> similar to org-agenda. The list is coming from the database and is
> blazing fast.
>
> Then I can filter using helm or ivy completion or built-in completion:
>
> - I can simply type name of person related to the task, be it myself
>   or somebody else. Majority of "my" tasks are related to other
>   people. I search by people, sometimes by companies or organizations.
>
> - I can type subject name or tag, any tag that need not be defined in
>   single Org file and I find the task
>
> - then if task is related to the person, I could quickly jump into
>   persons meta report, from where there is Org files, other files,
>   picture links all summarized in the meta Org profile, similar like
>   FBI profiles we see on movies, one find the person's name and can
>   see anything about that person.
>
> - or I could delete the task, re-assign, modify the task. It becomes
>   semi-automatically modified in the Org file of the user.
>
> Can I automated the execution of Babel code upon opening of the Org
> file?
>
> When all tasks become centralized by any means, in my case in the
> central database, then Org files get liberated from tasks and become
> structural and relational. I can edit each task and I can always see
> the updated list of the tasks and relations from one object to the
> other. Hyperlinks get automatically created.
>
> Personal problem is that tasks are sparse and separate in various Org
> files and not centralized. I become dependent of org-agenda to do what
> I need but it never does what I need.
>
> Example is the need to relate task to people. Tasks are people related.
>
> Purchasing cloths for your daughter? People.
>
> Visiting museum? People related.
>
> Some people are in Kenya, I do not think always on them. But I might
> when I come there. Then I would write "Kenya" to get people I need to
> put attention on. If tasks are centralized I can quickly jump to list
> of Kenyan people.
>
> If tasks are not centralized I need to put some tag under all those
> people's files "Kenya" which is repetitive and error prone activity.
>
> And I get trapped into "Org mode" for years. Which I am now, I am
> trapped but it does not help me to manage things how I think.
>
> I do value org-agenda but it is large effort to make SQL query out of
> the Org mode. And very expensive effort as it is huge program:
>
> -rw-r--r-- 1 415K Oct 19 10:20 org-agenda.el
>
> If I organize all my tasks centrally and only display them in Org
> files related to people, then making similar features like org-agenda
> becomes trivial.
>
> - Agenda for current week becomes trivial SQL query such as to select
>   those tasks by current week. Result becomes a list that may be used
>   either in Emacs completion or tabulated list mode or in Org or any
>   kind of modes.
>
> - To list entries with any kind of tags also become trivial, at least
>   for centralized database tasks that have been tagged.
>
> - searching for keywords is trivial single line SQL query.
>
> - multi-occur is not trivial, as that would involve searching in all
>   Org files. It asks for indexing Org files and going over them.
>
> - Finding any FLAGGED entries would become trivial
>
> It opens plethora of various means of reporting and accessing various
> action related notes or tasks.
>
> By general principle and due to nature of tasks that they do need
> attention, I think that all tasks should be centralized, any how, it
> does not matter how. From that principle I find it better that
> majority of tasks are handled in just few files and not many files as
> it becomes complex and users cannot any more remember which files.
>
> org-agenda and Org try to remember that for user, but there are many
> ways how Org files can be left without attention.
>
> If my staff member in other country uses Emacs such can access
> database and see assigned tasks. I can also export centralized tasks
> for the user and send such to user as Org file.
>
> If user updates tasks, as long as hyperlinks in the user's Org file
> are not disturbed, I can review the work of the user and just click on
> the hyperlink to update such task or mark them as DONE. Opinion of
> staff member on what is done and what is not done is not necessarily
> my opinion.
>
> Description of content body of the heading can be programmatically
> captured and entered into centralized database as a task.
>
> Then we comes to actual execution of tasks. How do we get reminded?
>
> Is the reminder only if I press {C-c a} for org-agenda? Do I need to
> do action to get reminded?
>
> I think computer shall be programmed to remind me on the start. Tasks
> that are in the database can be shown directly in Emacs, but SQL
> queries can be run and shown upon first log in into computer. Emails
> could be sent, SMS could be sent to remind me or other people assigned
> to the task. This brings attention and where people put attention that
> is becoming reality.
>
> Tasks could be shown by using timer in the mini buffer to remind user
> of what has to be done and what is next, what is today to be done.
>
> Keys can be assigned to TODAY, WEEK, MONTH, YEAR.
>
> Birthdays can be consolidated automatically, as if there is database
> of people, then their birtdays are automatic types of notes or tasks.
>
> People who travel often would like to see other people in the city. By
> changing one's location one could get list of known people and places
> in the city. If I am in Mombasa, Kenya I may forget that good
> restaurant as there are not many and visitor has to be brought in good
> one, not bad one (which there are many).
>
> With the introduction of emacs-libpq into GNU ELPA such features may
> become reality.
>
> emacs-libpq @ Github
> https://github.com/anse1/emacs-libpq
>
> Summary:
>
> - tasks are people related
>
> - plethora of Org files makes hard life to developers trying to satisfie users' needs
>
> - tasks should be centralized and related to objects such as people, organizations, subjects
>
> - Org files may be used as viewers for centralized task systems with
>   all the features left as they are as all properties and tags and
>   anything can be obtained from the database. Nothing changes.
>
> - upon saving Org file or upon click or invocation, the task in the
>   database can be updated if some description has been changed, but
>   updates can take place in the database directly.
>
> - tasks get liberated from any limits and formats, they can be just
>   hyperlinks or hyperdocuments to just anything.
>
> - integration and relation to many other objects becomes easier possible
>
> - searching, indexing, making plans, or creating new meta Org files on
>   the fly by using SQL queries becomes trivial. org-agenda of 450k can
>   become easily redundant
>   
> - once centralized, there must be reminding system that works within
>   or without Emacs, tasks are accessed by blazing speed, related
>   people's files are quickly located, various queries and ways of
>   reporting tasks in the list or meta Org file on the fly becomes
>   possible.
>
> I use "meta Org file" only to say that format is useful for displaying
> structured and relational information for the reason that it has
> hyperlinking support. As long as hyperlinks can be shown in any mode
> then reports could be displayed any how. 

Re: One vs many directories

[Jean Louis]

* Ihor Radchenko <yantar92@gmail.com> [2020-11-26 06:34]:
> > Can I automated the execution of Babel code upon opening of the Org
> > file?
> 
> Adding to other suggestions, you can always add a custom function to
> org-mode-hook instead of playing with file-local variables.
> 
> > Then we comes to actual execution of tasks. How do we get
> > reminded?

Yes, that requires customization, file local variables customize
things without user's customization. In the end my use for that slowly
disappears as I am transitioning all tasks to HyperScope. Yesterday I
made simple hard-coded function here that I invoke inside of an Org
heading that captures the heading, date created, and its text and
stores it as a note in the database.

(defun hyperscope-capture-org-heading ()
  "Captures Org heading and stores it in the Hyperscope dynamic
  knowledge repository"
  (interactive)
  (let* ((body (org-copy-subtree nil nil nil t))
	 (body (pop kill-ring))
	 (body (org-no-properties body))
	 (heading (org-get-heading))
	 (created (org-property-values "CREATED"))
	 (date (if created (substring (car created) 1 11) nil))
	 (body (with-temp-buffer
		 (insert body)
		 (org-mode)
		 (org-back-to-heading)
		 (kill-line)
		 (delete-matching-lines ":PROPERTIES:")
		 (delete-matching-lines ":CREATED:")
		 (delete-matching-lines ":ID:")
		 (delete-matching-lines ":END:")
		 (buffer-string))))
    (hyperscope-add-note-to-set heading body date)))

The other use I had for users are tables as I have been writing
tables so much. 

But then I have to write that Joe Doe have paid money to Jane Dane in
the file of Jane Dane as only so I know how much money Jane Dane keeps
on behalf of business. And then I have to write by hand the same
transaction in the file of Joe Doe, as now he has less money. Tracking
it by head is definitely after some time error prone activity.

Database already has few tables for accounts and businesses, so I can
use database backed accounting which is already implemented as journal
package. 

Then if I am doing transaction from Joe Doe I would select Joe Doe
petty cash acccount transferring money to Jane Dane's petty cash
account. I would type less and be less worried about errors. Entries
are stored in the database. If account for Joe Doe is related to Joe
Doe (relations has to be assigned, not just named by person) then I
can invoke the creation of Org table for the Joe Doe's profile.

Majority of times I am creating Org file on the fly for the person
that contains:

* Tasks
* Transactions

Now I will be pushing tasks into meta level and edit them from there
and when Org file is opened that relates to one person then Tasks can
automatically appear in the list ordered by their status or other
attribute.

Transactions I will most probably slowly transition into the database
and they can also automatically appear in the Org table under
Transactions so to be sent easier by using nice Org exports. 

While this approach may not be common, it offers me more free time and
less worries.

> > Is the reminder only if I press {C-c a} for org-agenda? Do I need to
> > do action to get reminded?
> 
> You can always configure Emacs to run agenda on startup. Just add a
> command to your init file ;)

Thank you.

> For automatic reminders, there is built-in org-notify.el or external
> org-alert package (https://github.com/spegoraro/org-alert).

That is definitely good idea. Just little limited as it relates to
user on computer and not to users to which tasks are assigned to.

Think of that as the next and long forgotten enhancement for Org. 

I have this property in Org header:

#+PROPERTY: ASSIGNED_ALL Ezekiel Jean James Victor Dejan TeamTZ

TeamTZ is property of several people, when task is assigned to TeamTZ,
such task get sent to all people involved.

When I am assigning tasks the above list is using one of them to be
assigned.

When task is assigned to somebody notification of a deadline or alerts
are definitely useful for me. But they are are useful for those
conducting the tasks.

Thus integration to remind those *related* people to the assigned
tasks and their deadlines or schedules would be useful.

- org-send-task-by-email (shows some properties, schedules, ID)

- org-send-task-by-email-ascii (not same display in email)

- org-send-task-by-sms (using KDE connect, Termux, gnokii, Twilio, etc)

- org-send-task-by-fax (using email2fax providers or built-in fax modem)

Then think of reminders, they would need to run maybe in Emacs, but
maybe in Emacs that runs from cron job periodically, whatever is more
reliable. 

I also think that generic reminding database or one `reminders' table
would be usable on every system so that users may add to it anything,
be it task, birthday, anniversary, deadlines, etc. Such generic
database would be run as system service and remind not only user on
computer buy outside people by using various connections such as SMS,
email, fax, notifications on computer, notifications on mobile
devices, XMPP chat, social networking and so on. Generic separate
database would be fed by outside programs such as Org, calendar,
external calendars, and so on.

> > Personal problem is that tasks are sparse and separate in various Org
> > files and not centralized. I become dependent of org-agenda to do what
> > I need but it never does what I need.
> 
> I agree that org-agenda has many issues that cannot be easily solved
> because of its complexity. However, everything you describe (including
> multi-occur) can also be achieved with org-ql
> (https://github.com/alphapapa/org-ql) - analogue of SQL query language
> for org-mode (with more optimisations in comparison with org-agenda). 

Definitely good only too much low-level. Users need finished functions
that integrate stuff. To adopt myself to org-ql would mean to read
documentations, meanings, and starting with some functions, while this
may be possible for me that approach is not user friendly as users
need integration and accessible interfaces. 

Example of lack of integration would be to tell to user to simply
construct the link in the form: [[Here][Link]] and that alone would
require some thinking and learning. 

Example of medium integration is when users are advised to press C-c
C-l,then they can choose type of the link and extend it and describe
the link.

Example of high level, best type of integration is Org features to
capture hyperlinks for example from eww browser or from rmail mail
reader like message ID. I did not try those as I have my own, but I
did read that such packages exist in the Org distribution and I find
THAT ideal scene and how it should be. Another good example are
browser bookmarklets for org-capture.

So the package org-ql in my opinion requires something like code
generators and wizards. While great for author, by installing that
package user cannot begin with it practically or in straight manner.

Good integration for org-ql would be a wizard that can add to the list
of various queries and offers users various ways to search such as
searching by headline, tags, or having both tags involved, date,
deadlines and so on. Then such list can be displayed automatically as
a menu or as completing read. That would be good high level
integration for org-ql. Something like how M-x imenu-list works could
be good integration. Program can recognize already all the tags and
offer to user a checklist, user could choose 1 or 2 or more tags to
get a subset of various agenda choices, or the function
imenu-add-to-menubar that generates Emacs menu with references to all
functions in Emacs Lisp, probably in other languages. Then org-ql can
be used as underlying system but higher level helpful functions would
assist in generating various queries, for user not transparent and not
something to think of.

Re: One vs many directories

[Ihor Radchenko]

Jean Louis <bugs@gnu.support> writes:

> When task is assigned to somebody notification of a deadline or alerts
> are definitely useful for me. But they are are useful for those
> conducting the tasks.
>
> Thus integration to remind those *related* people to the assigned
> tasks and their deadlines or schedules would be useful.

I think that can be done by extending alert.el package (which org-alert
is based on). alert.el allows the user to define custom alert styles,
including sending sms or email.

>> I agree that org-agenda has many issues that cannot be easily solved
>> because of its complexity. However, everything you describe (including
>> multi-occur) can also be achieved with org-ql
>> (https://github.com/alphapapa/org-ql) - analogue of SQL query language
>> for org-mode (with more optimisations in comparison with org-agenda). 
>
> Definitely good only too much low-level. Users need finished functions
> that integrate stuff. To adopt myself to org-ql would mean to read
> documentations, meanings, and starting with some functions, while this
> may be possible for me that approach is not user friendly as users
> need integration and accessible interfaces. 

Not really. Searching in org-ql is made very intuitive - just like you
would search in google. There are multiple interactive commands:
- helm-org-ql :: The user simply types search keywords and the commands
  shows all the matching headlines. For example, user can type
  "headline:Doe call todo:WAIT tags:invoice" and get all kinds of
  matches as he/she types
- org-ql-view :: shows the menu for different pre-customised searches
  like "Calendar: This week" or "Overview: NEXT tasks"

> Example of lack of integration would be to tell to user to simply
> construct the link in the form: [[Here][Link]] and that alone would
> require some thinking and learning. 

There is also (slower) helm-org (https://github.com/emacs-helm/helm-org)
offering similar interface to helm-org-ql, but also adding an option to
do various actions with the selected tasks, like inserting correctly
formatted link. The same can be done in helm-org-ql. I think the author
plans to extend that command in the near future. Or one can add extra
actions as needed - it is trivial to do in helm.

> Good integration for org-ql would be a wizard that can add to the list
> of various queries and offers users various ways to search such as
> searching by headline, tags, or having both tags involved, date,
> deadlines and so on.

All these are already available. The user can search tags:tag1,tag2,...,
headline:word1,word2 (or shortly hl:word1,...), scheduled:on=today,
deadline:from=-7,to=2020-12-01

Best,
Ihor

Re: One vs many directories

[Jean Louis]

* Ihor Radchenko <yantar92@gmail.com> [2020-11-29 10:52]:
> Jean Louis <bugs@gnu.support> writes:
> 
> > When task is assigned to somebody notification of a deadline or alerts
> > are definitely useful for me. But they are are useful for those
> > conducting the tasks.
> >
> > Thus integration to remind those *related* people to the assigned
> > tasks and their deadlines or schedules would be useful.
> 
> I think that can be done by extending alert.el package (which org-alert
> is based on). alert.el allows the user to define custom alert styles,
> including sending sms or email.

Yes, that is great.

We need more integration. There are low level packages but we need
integration, high level, user accessible and for users useful
features.

Emacs is itself useful but there may be long process until user starts
doing some programming. Then it becomes trivial to alert self or othre
people by SMS, yet majority of users will not have that functionality
unless we make it somehow.

org-set-reminder by SMS could send it to assigned person. This is for
me so important that it deserves menu item where user can assign that
reminder. Once assigned it would run in intervals until task has been
closed.

org-send-task-by-email would find out which user is assigned to
(feature is not built-in) and send it to assigned user
automatically. It is trivial, but we do not have that. One click and
it should go. 

> >> I agree that org-agenda has many issues that cannot be easily solved
> >> because of its complexity. However, everything you describe (including
> >> multi-occur) can also be achieved with org-ql
> >> (https://github.com/alphapapa/org-ql) - analogue of SQL query language
> >> for org-mode (with more optimisations in comparison with org-agenda). 
> >
> > Definitely good only too much low-level. Users need finished functions
> > that integrate stuff. To adopt myself to org-ql would mean to read
> > documentations, meanings, and starting with some functions, while this
> > may be possible for me that approach is not user friendly as users
> > need integration and accessible interfaces. 
> 
> Not really. Searching in org-ql is made very intuitive - just like you
> would search in google. There are multiple interactive commands:
> - helm-org-ql :: The user simply types search keywords and the commands
>   shows all the matching headlines. For example, user can type
>   "headline:Doe call todo:WAIT tags:invoice" and get all kinds of
>   matches as he/she types
> - org-ql-view :: shows the menu for different pre-customised searches
>   like "Calendar: This week" or "Overview: NEXT tasks"

That is game changer.

> 
> > Example of lack of integration would be to tell to user to simply
> > construct the link in the form: [[Here][Link]] and that alone would
> > require some thinking and learning. 
> 
> There is also (slower) helm-org (https://github.com/emacs-helm/helm-org)
> offering similar interface to helm-org-ql, but also adding an option to
> do various actions with the selected tasks, like inserting correctly
> formatted link. The same can be done in helm-org-ql. I think the author
> plans to extend that command in the near future. Or one can add extra
> actions as needed - it is trivial to do in helm.
> 
> > Good integration for org-ql would be a wizard that can add to the list
> > of various queries and offers users various ways to search such as
> > searching by headline, tags, or having both tags involved, date,
> > deadlines and so on.
> 
> All these are already available. The user can search tags:tag1,tag2,...,
> headline:word1,word2 (or shortly hl:word1,...), scheduled:on=today,
> deadline:from=-7,to=2020-12-01

org-ql is indeed a wizard for searching agenda and looks better than
default.

Jean

Re: One vs many directories

[Jonathan McHugh]

Texas,

Ive been developing a paradigm over the years based upon a (recursive) 6x6 grid
system, aligned with keybindings around the home row. Called Qiuy, I
refer to it as a "Recursive Modelling Language", given the annotations
work down to the level of functions and parameters, as well as directory naming.

While migrating my OS (Debian -> Guix) and Editor (Vim -> Emacs) has
muddied my workflows, I hope that my explorations with Org-Mode still
permit me to operate with a highly broad but shallow folder
structure. In any case, Im happy to be using Helm for browsing rapidly
(as you will understand reading below!).

Below is my home folder directory structure, please forgive legacy approaches clouding the
vision (Ive been eating my dogfood for a longtime, I will be ending
parental leave, hopefully the cruft and clutter can be trimmed/attic'd).

Should you have any interest I am drafting some of my thoughts re Qiuy,
as part of attendance at the upcoming Constant workshop
Bureaucracksy. Id be happy to send you some material if you wish.

=====================
HOME FOLDER DIRECTORIES:
=====================
1q10fq_music
1q10hqh_parsing
1q10iqi_authorship
1q10iqi_employment
1q10iqi_history
1q10iqi_initiating
1q10iqi_loops
1q10iqi_permutations
1q10iqi_states
1q10kq_bibliography
1q10rer_events
1q10rqr_coding
1q10rqr_tasks
1q10xq_etl
1q20beb_downloading
1q20be_bookmarks
1q20dq_appending
1q20dqd_document-management-systems
1q20dqd_editing
1q20dqd_file-managers
1q20dqd_removing
1q20dqd_text-editors
1q20dw_appending
1q20fe_images
1q20fq_diagrams
1q20fq_graph-plots
1q20fq_images
1q20fq_slides
1q20gq_fonts
1q20heh_parsing
1q20hqh_parsing
1q20iei_loading
1q20iqi_opening
1q20iqi_version-control
1q20jq_tables
1q20jq_templating
1q20ke_csv
1q20ke_html
1q20ke_json
1q20kq_csv
1q20kqk_document-management-systems
1q20kq_tables
1q20mq_contacts
1q20mq_content-types
1q20mq_document-body
1q20nqn_calendars
1q20oqo_classes
1q20rer_playing
1q20rq_analysing
1q20rq_calculating
1q20rq_comparing
1q20rqr_interviews
1q20tqt_curriculum-vitaes
1q20twt_notes
1q20ueu_searching
1q20uw_saving
1q20xq_converting
1q20xq_regular-expressions
1q20xq_substituting
1q20xqx_compression
1q20xqx_concatenating
1q20xqx_converting
1q20xqx_regex
1q20xqx_serialisation
1q30dqd_terminals
1q30fq_diagrams
1q30gq_backgrounds
1q30gq_color
1q30gq_fonts
1q30gqg_arcs
1q30gqg_arrows
1q30gqg_shapes
1q30gqg_theming
1q30gq_style
1q30iqi_prettifying
1q30iqi_states
1q30iqi_transparency
1q30jq_design
1q30jq_layouts
1q30jq_tables
1q30jq_templates
1q30mq_body
1q30mq_charts
1q30mq_graphs
1q30nq_networks
1q30uq_menus
1q30uw_outputting
1q30vq_backgrounds
1q30vq_columns
1q30vq_layers
1q30vq_monitors
1q30vq_panels
1q30vqv_graphics
1q30vqv_layouts
1q30vqv_statuslines
1q30vqv_window-managers
1q30vqv_windows
1q30xq_transforming
1q30yq_prettifying
1q30yq_visualising
1q40beb_scraping
1q40bqb_downloading
1q40bqb_networking
1q40bqb_servers
1q40bqb_transferring
1q40gqg_displaying
1q40gqg_theming
1q40hqh_coroutines
1q40hqh_scraping
1q40hq_navigating
1q40iqi_accessing
1q40jq_settings
1q40nqn_pathways
1q40nqn_positioning
1q40ueu_searching
1q40uq_exporting
1q40uq_importing
1q40uqu_searching
1q40yq_rotating
1q50cqc_terminals
1q50fe_files
1q50gqg_displaying
1q50hqh_interpreters
1q50hqh_parsing
1q50hq_navigating
1q50iqi_arguments
1q50iqi_building
1q50iqi_capturing
1q50iqi_contacts
1q50iqi_coroutines
1q50iqi_dates
1q50iqi_indexing
1q50iqi_keyboards
1q50iqi_loading
1q50iqi_parameters
1q50iqi_queues
1q50iqi_relationships
1q50iqi_states
1q50iqi_switching
1q50iqi_timing
1q50iqi_users
1q50iqi_vacancies
1q50iq_mappings
1q50iwi_terminating
1q50jq_manifests
1q50jq_options
1q50jq_stacks
1q50kek_tables
1q50mq_languages
1q50nqn_pathways
1q50nqn_relations
1q50nqn_trees
1q50nq_relations
1q50oqo_commands
1q50rq_analysing
1q50rq_calculating
1q50rq_graphs
1q50rq_identifying
1q50rq_rating
1q50rq_testing
1q50rq_watching
1q50tqt_curriculum-vitaes
1q50ueu_extracting
1q50uqu_filtering
1q50uqu_loading
1q50vqv_panels
1q50vqv_window-managers
1q50xq_companies
1q50xq_concatenating
1q60iei_loading
1q60jq_schemas
1q60jq_templating
1q60kek_databases
1q60me_classes
1q60me_schemas
1q60me_structure
1q60mq_asdls
1q60mq_catalogues
1q60nq_languages
1q60nq_package-management
1q60ue_updating
2q10dq_text-editing
2q10hqh_travelling
2q10iq_bureacracy
2q10iqi_history
2q10iqi_programming
2q10iqi_scheduling
2q10iwi_arbitration
2q10rqr_conferences
2q10rqr_employment
2q10rqr_travelling
2q20
2q20bqb_smtp
2q20cqc_text-editing
2q20cqc_word-processors
2q20dqd_text-editing
2q20fe_audio
2q20fe_film
2q20fe_graphics
2q20fe_images
2q20fe_literature
2q20fe_music
2q20fw_graphics
2q20fw_photos
2q20hqh_parsing
2q20hqh_scraping
2q20hq_navigating
2q20iwi_history
2q20kq_json
2q20kqk_databases
2q20kqk_document-management
2q20rq_analysing
2q20tq_signatures
2q20xq_converting
2q20xqx_converting
2q30fe_icons
2q30gqg_visualisations
2q30mq_columns
2q30mq_tables
2q30yqy_emphasizing
2q40bqb_downloading
2q40bqb_smtp
2q40bqb_transferring
2q40bq_downloading
2q40bq_websites
2q40hq_navigating
2q40hq_orientating
2q40tqt_mails
2q40twt_mailouts
2q40ueu_searching
2q50cqc_command-lines
2q50cqc_file-management
2q50dq_deployment
2q50dqd_generators
2q50dqd_interfacing
2q50dqd_interpreters
2q50dqd_terminal-multiplexers
2q50hq_navigating
2q50iqi_caching
2q50iqi_coroutines
2q50iqi_history
2q50iqi_security
2q50iq_sessions
2q50iwi_logs
2q50jqj_properties
2q50kqk_gis
2q50nen_cities
2q50nqn_pathways
2q50nq_properties
2q50nq_services
2q50rqr_activity
2q50rqr_history
2q50rqr_projects
2q50tq_data
2q50tq_logging
2q50tqt_profiles
2q50uq_searching
2q50vq_panels
2q50vqv_window-managers
2q5oiwi_archives
2q60iq_computer-science
2q60iq_finance
2q60iqi_security
2q60iqi_updating
2q60iq_policy
2q60iq_science
2q60kqk_content-management-systems
2q60kqk_structure-data
2q60mq_databases
2q60mq_dictionaries
2q60mq_information-systems
2q60mq_languages
2q60mqm_multi-stage-programming
2q60nq_operating-systems
2q60rqr_training
3q10iq_finance
3q10iq_information-systems
3q10rqr_building
3q10rqr_employment
3q10rqr_events
3q10rqr_side-projects
3q10tqt_proposals
3q20bw_websites
3q20cq_image-viewers
3q20dqd_text-editors
3q20fe_viewing
3q20fq_3d-printing
3q20fq_documents
3q20fq_files
3q20fq_logos
3q20fq_music
3q20fw_music
3q20gqg_design
3q20gqg_graphs
3q20heh_parsing
3q20kqk_content-management-systems
3q20kqk_document-management-systems
3q20kqk_food
3q20kqk_screencasts
3q20mq_characters
3q20mq_content-types
3q20re_textual-analysis
3q20tq_dictionaries
3q20tqt_social-media
3q20twt_reporting
3q20twt_submissions
3q20ueu_aggregating
3q30fq_fonts
3q30gq_color
3q30gq_colors
3q30gqg_diagrams
3q30gqg_styles
3q30kqk_fields
3q30rqr_employment
3q30tqt_curriculum-vitaes
3q30tqt_displaying
3q30uqu_elipses
3q30uqu_orbits
3q30vq_visualisation
3q30vq_window-managers
3q30yqy_emphasizing
3q40bqb_browsers
3q40bqb_content-delivery
3q40bqb_exporting
3q40bqb_mailing-lists
3q40bqb_transferring
3q40fq_emails
3q40hq_orientating
3q40nqn_geodata
3q40ueu_searching
3q50bq_social-media
3q50bq_websites
3q50cqc_shells
3q50dqd_shells
3q50fq_colors
3q50gq_color
3q50gq_colorschemes
3q50gqg_colorschemes
3q50gqg_computer-aided-design
3q50hq_orientating
3q50iq_electricity
3q50iqi_authorship
3q50iqi_duplicates
3q50iqi_performance
3q50iqi_terminal-multiplexers
3q50iqi_usability
3q50iq_management
3q50iq_sustainability
3q50iq_ventilation
3q50jq_configuring
3q50jqj_buildings
3q50jq_profiles
3q50jq_snippets
3q50jq_theming
3q50kqk_keyboards
3q50mq_syntax
3q50nq_calendars
3q50rq_analysing
3q50rqr_maintenance
3q50rqr_programming
3q50rq_troubleshooting
3q50tq_data
3q50tqt_bookmarks
3q50tqt_messaging
3q50tq_variables
3q50uq_orientating
3q50yqy_compositors
3q60be_websites
3q60dq_keyboards
3q60iq_application-modelling
3q60iq_application-modelling-and-design
3q60iq_automation
3q60iq_informatics
3q60iq_information-society
3q60iqi_permissions
3q60iq_outsourcing
3q60iq_politics
3q60iq_security
3q60iq_storage
3q60iq_utilities
3q60jqj_healthcare
3q60kqk_content-management-systems
3q60mq_languages
3q60nq_package-management
3q60oqo_software
3q60rq_glossaries
3q60rq_ontologies
3q60rqr_elections
3q60rq_semantics
3q60rq_taxonomy
4q10iqi_scheduling
4q10rqr_events
4q10rqr_language-classes
4q20be_bookmarks
4q20iqi_synchronising
4q20ke_xml
4q30fq_fonts
4q40bqb_ftp
4q40bqb_irc
4q40bqb_rss
4q40bqb_smtp
4q40bqb_torrenting
4q40bq_servers
4q40cqc_web-browsing
4q40hq_connecting
4q40iqi_accessing
4q40nqn_pathways
4q40ue_searching
4q40uq_clients
4q50be_bookmarks
4q50bqb_transferring
4q50bqb_vehicles
4q50dqd_bindings
4q50fqf_contact-books
4q50iqi_access
4q50iqi_accessing
4q50iqi_sockets
4q50jq_aliases
4q50nqn_locations
4q50nqn_pathways
4q50nq_security
4q50tq_repositories
4q50tqt_curriculum-vitaes
4q50tqt_notifications
4q50uw_exporting
4q60iqi_access
4q60iqi_certificates
4q60iqi_passwords
4q60iqi_permissions
5q10dq_interpreters
5q10iqi_autostarting
5q10iqi_birthdays
5q10iqi_completing
5q10iqi_postponing
5q10iqi_scheduling
5q10iqi_sessions
5q10iqi_states
5q10iqi_tasks
5q10iqi_workflows
5q10nq_employers
5q10rqr_activity
5q10rqr_events
5q10rqr_helping
5q10rqr_tasks
5q10rqr_writing
5q10tqt_memorandums
5q10tqt_reminders
5q10tw_history
5q10twt_curriculum-vitaes
5q10twt_education
5q10twt_employment
5q10xq_refactoring
5q20bqb_networks
5q20cec_image-viewers
5q20cqc_document-viewers
5q20cqc_media-players
5q20cqc_music-players
5q20cqc_pdf-viewers
5q20dq_altering
5q20dq_completing
5q20dqd_cad
5q20dqd_document-editors
5q20dq_deleting
5q20dqd_text-editors
5q20dqd_video-editors
5q20dq_removing
5q20fe_icons
5q20fq_compact-discs
5q20fq_flags
5q20fq_images
5q20fq_logos
5q20fq_media
5q20fq_music
5q20hqh_extracting
5q20hqh_parsing
5q20iqi_access
5q20iqi_drafting
5q20iqi_loading
5q20iqi_revisions
5q20jq_metadata
5q20kq_formats
5q20kq_html
5q20kqk_content-management-systems
5q20kqk_databases
5q20kqk_document-management-systems
5q20kqk_file-managers
5q20kqk_system-tools
5q20kqk_templating
5q20kq_type-file
5q20mq_boxes
5q20mq_dictionaries
5q20nen_directories
5q20nq_operating-systems
5q20rq_spelling
5q20ueu_searching
5q20uqu_searching
5q20xq_refactoring
5q20xqx_compression
5q20xqx_copying
5q20xqx_encoding
5q30cqc_terminal-multiplexers
5q30dqd_terminals
5q30dq_interfaces
5q30fq_fonts
5q30fq_logos
5q30fq_wallpapers
5q30gq_coloring
5q30gq_colorschemes
5q30gqg_alignment
5q30gqg_sizes
5q30gqg_spacing
5q30gqg_venn
5q30gq_layouts
5q30iqi_sorting
5q30jq_themes
5q30kqk_terminals
5q30mq_frames
5q30mq_headers
5q30mq_panels
5q30nqn_positioning
5q30nqn_spacing
5q30uw_outputting
5q30vq_desktop-environments
5q30vq_panels
5q30vqv_desktop-environments
5q30vqv_panels
5q30vqv_terminal-environments
5q30vqv_window-managers
5q30vq_windows
5q30yq_emphasizing
5q30yqy_opacity
5q40bqb_ftp
5q40bqb_moving
5q40bqb_smtp
5q40bqb_transferring
5q40bqb_voip
5q40bq_transferring
5q40cqc_browsers
5q40cqc_interfaces
5q40cqc_shells
5q40cqc_terminal-multiplexers
5q40fe_wallpapers
5q40fq_hyperlinks
5q40fq_websites
5q40hq_navigating
5q40hq_orientating
5q40iqi_terminal-multiplexing
5q40kqk_hooks
5q40kqk_mail-clients
5q40nqn_alignment
5q40nqn_directories
5q40nqn_pathways
5q40nqn_urls
5q40tqt_mails
5q40tqt_messaging
5q40ueu_searching
5q40uq_notifications
5q40uq_orientating
5q40uq_receiving
5q40uq_sending
5q40vqv_windows
5q40yq_coordinates
5q50dq_bindings
5q50dq_commanding
5q50dq_completing
5q50dqd_editing
5q50dqd_selecting
5q50dq_editing
5q50dq_interacting
5q50dq_interpreters
5q50hq_hooks
5q50iei_loading
5q50iqi_building
5q50iqi_cache
5q50iqi_contexts
5q50iqi_dates
5q50iqi_indexing
5q50iqi_interacting
5q50iqi_local
5q50iqi_reloading
5q50iqi_revision-management
5q50iqi_sessions
5q50iqi_shells
5q50iqi_states
5q50iqi_variables
5q50iqi_volume
5q50jq_configuring
5q50jq_dotfiles
5q50jq_metadata
5q50jq_templating
5q50mq_language
5q50nq_contacts
5q50nq_home-folders
5q50nqn_pathways
5q50nqn_sourcing
5q50nq_relations
5q50oqo_variables
5q50rq_analysing
5q50rq_debugging
5q50rq_duplicates
5q50rq_monitoring
5q50rq_testing
5q50rq_usability
5q50tq_caching
5q50tq_history
5q50tq_metadata
5q50ue_sourcing
5q50uq_orientating
5q60cqc_terminal-multiplexers
5q60hq_orientating
5q60iqi_access
5q60iqi_initiating
5q60iqi_permissions
5q60iqi_security
5q60iqi_states
5q60iqi_virtualizing
5q60jq_accounts
5q60jq_profiles
5q60jq_settings
5q60jq_templating
5q60kq_file-types
5q60kq_formats
5q60kqk_content-management-systems
5q60kqk_databases
5q60mq_languages
5q60mq_structure
5q60mq_structures
5q60nq_operating-systems
5q60nq_package-management
5q60nq_packages
5q60oqo_libraries
5q60oqo_macros
5q60oqo_tools
5q60xqx_compiling
6q10nqn_calendars
6q10rqr_events
6q10rqr_tasks
6q10xq_converting
6q20
6q20bqb_browsers
6q20bqb_printers
6q20bqb_social-media
6q20cqc_file-managers
6q20dqd_functional-programming
6q20dqd_text-editing
6q20fq_books
6q20fq_computer-games
6q20fq_images
6q20fq_media
6q20fq_music
6q20fq_pdfs
6q20fq_videos
6q20hqh_parsing
6q20iqi_buffers
6q20iqi_version-control
6q20kq_json
6q20kqk_computer-aided-design
6q20kqk_content-management-systems
6q20kqk_databases
6q20kqk_document-management-systems
6q20kqk_image-handling
6q20kqk_pdf-management-systems
6q20kqk_text-editing
6q20mq_cover-letters
6q20mq_curriculum-vitaes
6q20oqo_classes
6q20rq_analysing
6q20rwr_recording
6q20tqt_attachments
6q20uqu_document-viewers
6q20xqx_compiling
6q20xqx_compression
6q20xqx_converting
6q20xqx_encoding
6q20xqx_refactoring
6q20xqx_regular-expressions
6q20xqx_serializing
6q30fe_pdfs
6q30gq_fonts
6q30gqg_colorschemes
6q30jq_design
6q30jq_templating
6q30jq_themes
6q30kqk_curses
6q30kqk_menus
6q30kqk_rendering
6q30kqk_terminals
6q30ue_displaying
6q30uw_outputting
6q30uw_window-management
6q30vqv_desktop-environments
6q30vq_visualisation
6q30vqv_panels
6q30vqv_statuslines
6q30vqv_window-managers
6q40bqb_browsers
6q40bqb_downloading
6q40bqb_messaging
6q40bqb_piping
6q40bqb_scraping
6q40bqb_servers
6q40bqb_voip
6q40hq_browsers
6q40hq_buffers
6q40hq_navigating
6q40hq_orientating
6q40hq_ssl
6q40kqk_asynchronous
6q40kqk_coroutines
6q40kqk_event-loops
6q40kqk_foreign-function-interfacing
6q40kqk_multithreading
6q40kqk_threading
6q40tqt_messaging
6q40tqt_notifying
6q40ueu_searching
6q50
6q50cqc_interpreters
6q50dq_completing
6q50dqd_erasing
6q50dqd_generators
6q50dqd_installers
6q50dqd_terminals
6q50iqi_building
6q50iqi_compiling
6q50iqi_history
6q50iqi_shells
6q50iq_utility
6q50kqk_environments
6q50kqk_shells
6q50kqk_version-control
6q50oqo_commands
6q50oqo_functions
6q50rq_monitoring
6q50rqr_administrating
6q50rqr_bindings
6q50rqr_touch-displays
6q50rq_testing
6q50rq_validating
6q50xq_converting
6q50xqx_converting
6q60iq_batteries
6q60iqi_access
6q60iqi_identity-cards
6q60iqi_permissions
6q60iqi_security
6q60iq_usability
6q60iq_utilities
6q60kqk_drivers
6q60kqk_firmware
6q60kqk_middleware
6q60kqk_operating-systems
6q60kqk_package-managers
6q60kqk_repos
6q60mq_language
6q60mq_languages
6q60mq_structures
6q60nq_operating-systems
6q60nq_package-management
6q60nq_packages
6q60nq_plugins
6q60rq_testing
6q_tools-qiuy
Desktop
Documents
Downloads
etc
Videos


=====
Kind regards,


Jonathan






Texas Cyberthal <texas.cyberthal@gmail.com> writes:

> Having a tall directory tree with many leaves and branches is against
> Org's philosophy.
>
> Here is my argument that such a structure is objectively correct for
> personal info management:
>
> https://github.com/cyberthal/10-Bins-template
>
> For the record, Org works fine with this, although I had to do a bit
> of config to get around the default philosophy.


-- 
Jonathan McHugh
indieterminacy@libre.brussels

Re: One vs many directories

[Jean Louis]

* Jonathan McHugh <indieterminacy@libre.brussels> [2020-11-21 16:43]:
> Texas,
> 
> Ive been developing a paradigm over the years based upon a (recursive) 6x6 grid
> system, aligned with keybindings around the home row. Called Qiuy, I
> refer to it as a "Recursive Modelling Language", given the annotations
> work down to the level of functions and parameters, as well as
> directory naming.

Explain details please.

How do you file stuff?

How do you retrieve it?

Side notes:

> While migrating my OS (Debian -> Guix)

Guix is fully free OS, GNU itself using GNU Shepherd init system with
scheme as system language.

> and Editor (Vim -> Emacs)

Vim is just fine and Emacs too. I never switched from vi to Emacs, I
use them both. And I use ed especially to edit system configurations
quickly. 

> has muddied my workflows, I hope that my explorations with Org-Mode
> still permit me to operate with a highly broad but shallow folder
> structure. In any case, Im happy to be using Helm for browsing
> rapidly (as you will understand reading below!).

I did not understand it. Need details.

> Should you have any interest I am drafting some of my thoughts re Qiuy,
> as part of attendance at the upcoming Constant workshop
> Bureaucracksy. Id be happy to send you some material if you wish.

Yes pleas, send it.

> =====================
> HOME FOLDER DIRECTORIES:
> =====================
> 1q10fq_music
> 1q10hqh_parsing

How do you use that?