Re: Adding Org Files to org-agenda-files

[Ihor Radchenko <yantar92@gmail.com>]

Dear Jean Louis,

Thank you for the detailed insight into your extensive experience of
project management and practical planning. I do not have that much
experience, but can provide a significantly different point of view
related to my research work.

Jean Louis <bugs@gnu.support> writes:

> * Ihor Radchenko <yantar92@gmail.com> [2020-12-01 05:21]:
>> Jean Louis <bugs@gnu.support> writes:
>
> Just as you got a hunch, random incidents happen all the time on
> ground. There is set of policies and staff members get trained to
> apply them. For example our coordination policy is to pretty much
> coordinate any reasonable action before, during and after
> execution. If staff member is departing to village such will send a
> message and we know what is the action of a staff member. If
> supervisor is on computer such action can be entered in same central
> file. Otherwise email list of staff member holds track of actions.
>
> In that sense we help each other.

Thanks for providing an example. I do agree that the management model
you are using for your job fits into defining projects rather strictly
and delegating the planning/non-trivial decision making to competent
people. In such a context, ordered project plans with a single action at
a time and each employee assigned to a single project do make a lot of
sense. However, different perspectives do exist.

My personal experience is doing a lot of research work. That's probably
on the other side of the spectrum from the environment you are working
in. I cannot define very concrete steps to execute a research project.
Not because it is impossible, but rather because failures are pretty
much guaranteed far before all the steps are executed. Moreover, most of
time, it is not possible to consult someone else on resolution of the
problem causing blockage, simply because the problem is something that
never ever appeared in the past (that's the whole point of doing
research). Instead, I need to spend a significant time trying to find
*similar* problems digging through literature, talking to people working
on related problems, or even just thinking. Then, waiting until the
solution appears becomes a waste of time (there is even no guarantee
that solution exists) - if there are other alternative approaches to
achieve the global project objective, they would better be tried before
the blockage in one particular direction in solved. In fact, switching
to alternative approaches (or even projects) sometimes help to look at
the problem from different angle and solve it. The described difficulty
is *underestimation* of what can happen - even the initial project
objectives can be changed according to the current research results.
Trying to stick to a strict project structure in such a situation is a
waste of time - project must be re-created from scratch very too often,
unless it is more flexible from the very beginning.

In fact, the situation does not apply to a single project. The whole
project can be stuck and it is often helpful to have multiple projects
that can be done (though it is necessary to stick to highest-priority
project when possible).

The described situation is where NEXT tasks/projects can become
extremely helpful. Multiple NEXT tasks do not mean that I need to look
at them every day and switch from one to another. There are NEXT tasks
and there are NEXT tasks that are actually scheduled on specific day.
One day cannot have more than several (ideally one) NEXT task (possibly
containing a checklist). That's where agenda comes handy. It is not used
to decide what to do during that day. It merely shows earlier decision
when planning which project (and corresponding doable NEXT task) to do
on specific day. Other items in agenda are things that must be done on
that day anyway (meetings, mandatory habits, etc). Polluting agenda with
unnecessary staff is no better than mindless browsing of youtube.

> After this discussion and review of how SMOS implemented NEXT and how
> some people implement NEXT while doing their planning with Org mode,
> I see that it will never be necessary on my side. Just never.
>
> This is for reason that we use set of policies beforehand and train
> people how to do projects. Number one is that person cannot start
> doing any action without fully understanding all parts of the full
> project. We expect person to be literate and capable at least in the
> context of the project being executed. We push the purpose of the
> project and reason, not the execution of single tasks. As purpose of
> tasks are to achieve the purpose, person executing those tasks is
> supposed to collaborate on the project and contribute to it. Executing
> tasks is done by reason and not by robotic planning.

> ...

> That should clearly answer why NEXT is completely redundant as in all
> experience of years of planning, writing projects and assigning such
> to people I have not even encountered a problem related to the subject
> "NEXT" as used by people in Org planning:
>
> - there is set of policies on how to train people for projects
>
> - there is set of policies how to coordinate, communicate, report,
>   including report on events
>
> - plans have goals and purposes, projects fulfill one step of a plan,
>   projects have its own purposes and tasks are there to complete a
>   project

> ...

I hope I described my use-case sufficiently to show the difference with
your situation. For research, "fully understanding all parts of the full
project" means that project is pretty much completed and there is no
need to look further except maybe writing reports.

As I mentioned earlier, the purpose of NEXT items is not for daily use.
That's where scheduling can be used (at least, in my workflow). The
purpose of NEXT items is making project review easier - they are mainly
needed to provide hints on decision how to proceed with a blocked
project. As you mentioned, this is useless when project steps are
well-defined and little trouble is expected during execution.

> - any task becomes reasonably redundant if we have achieved the
>   project target. Any project becomes redundant if plan's step or
>   plan's purpose have been achieved. This is contradictory to
>   robotic way of how Org have been programmed in relation to list
>   items:
>
>   - mark heading with TODO (let us say project purpose)
>   - [ ] add TODO list items
>   - only if all TODO list items are marked [X] the parent node can be
>     marked as DONE
>
>   That approach is contradictory to human logic of achieving things. I
>   am not doing a single task for the single task's sake but for higher
>   purpose and if higher purpose have been achieved, all those planned
>   single tasks become reasonably redundant.

If target is flexible (like in research), extra TODO items can be useful
as a reminder what else might be done. Also, note that org-mode does not
strictly force todo dependencies. One can always force unconditional
todo state change with C-u C-u C-u C-c C-t (or by setting
org-enforce-todo-dependencies and
org-enforce-todo-checkbox-dependencies).

>   I am using word "reasonably" as that involves human who decides
>   about it and not robotic following of the tasks and executing them
>   just because they may appear as not DONE.

I look at it from different perspective. Task dependency is forcing me
to double-check the tasks not marked done and explicitly thinking if I
need to do them and improve the project (remember, there is no
well-defined project goal for me - things can always be improved, unless
there is time limit). If I decide to not do the not-done task (by
actively thinking, not by mindlessly marking project done just because I
think the goals are nominally achieved), I just mark the task CANCELLED
(which is a type of "done" keywords in org terminology). At the end,
task dependency allows to double-check for any missing ideas I could
forget about.

> What is not reason is to have unreasonable files of allegedly ordered
> tasks which are in reality not ordered and proof for that is that
> org-agenda exists in the first place. People do not keep their
> projects and tasks in ordered manner and they need org-agenda.
>
> That is why I almost never used org-agenda in last 5 years.

While reading your examples about why org-mode is often promoting
procrastination and messed up organisation, I feel that you expect more
from org-mode than it is.

You provided examples that people used their brains instead of computers
and paper instead of files in the past and successfully managed complex
projects. I would like to point out that org-mode to organisation and
project management is just like pen and paper to project management and
organisation. It is easy to have paper notes scattered all around the
office, home, and half of them lost somewhere. Same in org-mode, and you
provided enough examples. One needs to have a proper mindset and
established workflows to manage real projects with pen and papers. I
think about org-mode as about improved pen and paper - with proper
workflows and organisation it can be very efficient; without
organisation - it's just a digital mess, worse than some computer
desktops. org-mode provides a set of instruments - they can be used in
vastly different project management styles, some are more suitable to
specific styles, some are less suitable. As you mentioned, org-agenda is
not suitable for your style. It can be much better for others.

> org-agenda may be useful but it is on bottom of things, not on top of
> things. Tasks in such planning do not belong anywhere, they are
> distributed among files that are named any how where people do not
> have any real method of sorting them. org-agenda will show then
> anything, from personal tasks to business tasks, recreational, family
> tasks or anything together and it does not make sense to me.

While agenda can certainly show such kind of mix, it is indeed very
inefficient use of this tool. If other readers of this thread are
interested in better practices on using agenda, I recommend what is
recommended in [1]. It is absolutely crucial to keep daily agenda as
small as possible - only tasks that must be done on that day *and in the
location context* should be shown. Mixture of home and work tasks must
not happen. I knew this when I just started playing around with GTD, and
I thought that it is not important. After years of experience, I have to
say, that the rules about agenda are determinal to finishing work that
matters.

[1] Allen David [2015] Getting things done : the art of stress-free productivity

> Working on Org file means working from bottom to top:
>
> - make tasks, little here, little there, organize maybe by some
>   groups, make this or that file, search through agenda because I have
>   not ordered anything how it should be. Think of task first because
>   it is scheduled for its own sake of being scheduled. Do the task
>   because it is task and not part of one higher purpose. Mark flag,
>   add properties, tag them to be able to search them.
>
> The Org way of doing things is organizing procrastination with more
> and more increasing complexities that are allegedly supposed to make
> life easier.
>
> Please do not stone me.

While one can work with org file the way you described, it is not
necessary (and should not be done most of the time). High-level planning
is very important. It can be ignored to capture ideas in the middle of
doing something else, but those captured ideas should be thought about
in context of the whole project and placed into (or discarded from) the
project according to top-level objectives.

> Here is structure of a project, as part of bigger plan. Projects can
> be structured any how on my side. When assigned to other people there
> are sections of introduction:
>
> 1 Primary principle for reading ;; explains to people not to skip misunderstoods
> 2 Primary principle for communication ;; that we shall collaborate, etc.
> 3 Definitions of words ;; defines terms related to project
> 4 About company
> 5 Goal of the project ;; known objective, actions are done to achieve
>                          the goal and it has clear quote
> 6 Purpose of the project ;; A purpose is a lesser goal applying to
>                             specific activities or the sujects. It
> 			    often expresses future intentions
> 7 Requirements for this project ;; no moving to "TODO" without it!
> 8 How to do this project ;; explains how to conduct project, reason,
>                             logic, collaboration is all here
> 9 How to report        
> 10 How to report on events
> 11 How to make pictures
> 12 Communication requirements [0/16] 
> 13 Personal introduction
> 14 Project steps            ;; this is where operational targets are defined
> 15 Awards

Note: This project template is fairly similar to what is recommended by
Allen David, except reporting and communication. I lack experience of
large collaborations, so cannot elaborate much on this part.

> If things are well organized from ground up then agenda becomes
> redundant.
>
> Organized implies to me to know what is next to be done.
>
> Unorganized person does not know what is next to be done. That is why
> Org agenda is there. Because tasks are scattered, not organized.

Agenda cannot help unorganised person. Similarly with a paper (or paper
calendar) that cannot help unorganised person. However, either calendar
or agenda can be used efficiently as tools helping organisation (when
they are suitable for the specific situation).

> Org mode has headings and hierarchy and established ways for people to
> order their goals, projects, tasks, but it is not what people are
> doing, because there is no form structure in Org mode to tell where
> something is allowed to be ordered and where not.

Well-organised person would not need computer to keep records in
relational database - even a simple paper would do if used properly [2].
org-mode provides such tools, but org-mode does not teach or enforce
organisation. The cost of being flexible is possibility to misuse. The
power of being flexible is possibility to use much more efficiently than
more restricted tools.

[2] https://www.lesswrong.com/posts/NfdHG6oHBJ8Qxc26s/the-zettelkasten-method-1

> Each planning methodology requires something names goals or purposes
> or objectives or targets and anything that has to be executed belong
> to such goals. In military they will call them objectives. Myself I do
> not approve of any wars neither military preparations, human animal is
> crazy. But military planning methodology does not involve any random
> searches over bunch of scattered tasks and data to find out what is
> scheduled, etc. Army, marines, government officers in many countries
> have methodology of planning that may be paper based or computer based
> and outperforms any type of discussed Org established ways of
> gathering the scattered.
>
> Thinking on long-range goal helps in determining short-range goals,
> which help in determining which projects or tasks are to be executed.

One can also refer to GTD methodology, which is more about long-term
goals than about individual task - the point many people miss. (Search
for GTD: Purpose, vision, goals, and areas of responsibility + weekly
review).

>> > children nodes with the tag. It becomes very trivial when using
>> > database with nodes having a parent:
>> >
>> > ,----
>> > | UPDATE hlinks SET hlinks_tags = 'TODO' WHERE hlinks_parent = THIS ONE;
>> > `----
>> >
>> > But rather a function would be used or type assigned. The above is
>> > only example that shows how complex hard coded Elisp functions can be
>> > replaced with 3-4 lines single function when database is a backend.
>> 
>> Why do you think that analogous Elisp function would be complex?
>> 
>> (defun yant/trigger-children (arg)
>>   "Change all the children to TODO when parent is TODO."
>>   (when (and (eq (plist-get arg :type) 'todo-state-change)
>> 	     (not (boundp 'trigger-children-progress))
>>              (string= (plist-get arg :to) "TODO"))
>>     (let (trigger-children-progress)
>>       (org-map-tree (lambda () (org-todo "TODO"))))))
>> (add-hook 'org-trigger-hook #'yant/trigger-children)
>
> Good for you, good for me. But not good as a product for people who
> are not programmers.

For people who are not programmers, the same can be done manually using
keyboard macro, which is even easier than a need to learn SQL (probably
because I don't know SQL and know macros).

Best,
Ihor