FR: headline iteration API

FR: headline iteration API

[Adam Spiers]

Hi all,

I would like to propose a new extensibility feature: an API for
iterating over all the headlines in a given buffer/file or set of
buffers/files.  In a manner similar to mapcar, this would allow you to
execute an elisp function for each headline which could do anything
you want.  It would *not* have any mandatory side-effects such as
modifying any buffer etc. - that would be left to the extension
developer to do.  Having said that, an API set of helper functions for
things like "increase the priority of this headline" or "add a tag to
this headline" would enrich the possible applications, although those
helper functions probably mostly exist already, even if they are not
explicitly documented as an API at the moment?

Anyway, example use cases:

  - Generating custom reports containing arbitrary combinations
    of things like the following:

    - (count of number of) items ...
      - grouped by TODO keyword state
      - grouped by tag
      - grouped by category
      - grouped by "area of responsibility"
      - grouped by priority
      - with a scheduled date/time
      - with a scheduled date/time in the past
      - with a scheduled date/time within a certain period
      - with a deadline
      - with a deadline missed
      - with a deadline within a certain period

    - amount of time scheduled for a particular type of
      TODO/tag/category/priority (e.g. to ensure that you are planning
      to spend your time evenly across various areas of
      responsibility, this would greatly assist with GTD-like
      "30,000ft and above" reviews)

    - actions taken during minuting of a meeting, grouped by owner

I am sure there are many other really cool ideas which the community
could come up with.  Reports could even be auto-generated via
`after-save-hook' or similar.

Obviously the agenda generation code has a lot of sophisticated
iteration capabilities already built in - without looking at the code,
I am wildly guessing that with some refactoring this logic could maybe
be reused, so that a new headline iteration API would support existing
mechanisms such as tag searches, org-agenda-skip, and sorting?  If so,
IMHO that would be really amazingly powerful :-)

Would be very interested in everyone's thoughts.  Thanks!
Adam

Re: FR: headline iteration API

[Daniel Clemente]

Hi,
  I find this very interesting. Some ideas:

- Maybe it could provide access not only to the headlines, but to the headlines of the headlines, and of their headlines, etc. In fact, access to the whole tree. This would allow batch processing of any contents. As a plus, each header would get a path (like XPath in XML, or DOM in XHTML) and would be addressable. Maybe useful for links.

- If there is an API to the content, applications can use data stored in org files in simple ways. For instance: „for all the headers with tag AAAA, get the the subheaders except if they are DONE, and get the property BBBB“. I don't know the „tree“ data structures in Emacs Lisp, but I suppose that representing a whole org file as a LISP object would be cool; also for easy exporting and data interchange.

- An API would be useful not only to generate reports, etc. like in your examples, but also to execute actions on each header; for instance „export“ each header, „delete“ it, ... (And now with latest version, each header can have different export options)

- Of course, this is already possible by going through the file looking for headers; an API would make iteration easier, like with mapcar 


  And many more things...


Daniel

Re: FR: headline iteration API

[Adam Spiers]

On Wed, Jun 04, 2008 at 04:35:01PM +0200, Daniel Clemente wrote:
> Hi,
>   I find this very interesting. Some ideas:
> 
> - Maybe it could provide access not only to the headlines, but to
>   the headlines of the headlines, and of their headlines, etc. In
>   fact, access to the whole tree. This would allow batch processing
>   of any contents.

Absolutely - that was a core part of the intent of my proposal; I
probably didn't make that clear enough.

> As a plus, each header would get a path (like XPath in XML, or DOM
> in XHTML) and would be addressable. Maybe useful for links.

Whilst borrowing some of the XPath syntax would be an extremely cool
way of making content addressable, I do think it's a nice-to-have
which is separate to having an iteration API.  Maybe later on the API
could be extended to only apply to subtrees matching an XPath-like
query, but that really would be icing on the cake.  I feel that the
biggest wins (and very big too, IMHO) would come from a simple
iteration API.

> - An API would be useful not only to generate reports, etc. like in your examples, but also to execute actions on each header; for instance „export“ each header, „delete“ it, ... (And now with latest version, each header can have different export options)

Agreed - it would open up many options for batch processing as well as
the reporting examples I gave before.  I would personally use it to
convert some of my tags into properties.  For instance I have tags
like 'sub60', 'sub2' to indicate that I estimate a task will take 60
minutes or 2 hours maximum, respectively.  They could be converted
into the effort estimation property.

> - Of course, this is already possible by going through the file looking for headers; an API would make iteration easier, like with mapcar 

Yes.  There will be many intricacies involved in iteration, and
probably only Carsten knows them all - without an API we would
individually be reinventing the same wheel over and over ... badly.

RE: FR: headline iteration API

[Dominik, C.]

[-- Attachment #1.1: Type: text/plain, Size: 2998 bytes --]


Hi Adam and others,

I do like the idea of an API to iterate of entries and outline trees.
For now, I am following this discussion to see what ideas pop up.
When I find the time, something will be implemented.

- Carsten

-----Original Message-----
From: emacs-orgmode-bounces+dominik=science.uva.nl@gnu.org on behalf of Adam Spiers
Sent: Fri 5/30/2008 2:46 PM
To: org-mode mailing list
Subject: [Orgmode] FR: headline iteration API
 
Hi all,

I would like to propose a new extensibility feature: an API for
iterating over all the headlines in a given buffer/file or set of
buffers/files.  In a manner similar to mapcar, this would allow you to
execute an elisp function for each headline which could do anything
you want.  It would *not* have any mandatory side-effects such as
modifying any buffer etc. - that would be left to the extension
developer to do.  Having said that, an API set of helper functions for
things like "increase the priority of this headline" or "add a tag to
this headline" would enrich the possible applications, although those
helper functions probably mostly exist already, even if they are not
explicitly documented as an API at the moment?

Anyway, example use cases:

  - Generating custom reports containing arbitrary combinations
    of things like the following:

    - (count of number of) items ...
      - grouped by TODO keyword state
      - grouped by tag
      - grouped by category
      - grouped by "area of responsibility"
      - grouped by priority
      - with a scheduled date/time
      - with a scheduled date/time in the past
      - with a scheduled date/time within a certain period
      - with a deadline
      - with a deadline missed
      - with a deadline within a certain period

    - amount of time scheduled for a particular type of
      TODO/tag/category/priority (e.g. to ensure that you are planning
      to spend your time evenly across various areas of
      responsibility, this would greatly assist with GTD-like
      "30,000ft and above" reviews)

    - actions taken during minuting of a meeting, grouped by owner

I am sure there are many other really cool ideas which the community
could come up with.  Reports could even be auto-generated via
`after-save-hook' or similar.

Obviously the agenda generation code has a lot of sophisticated
iteration capabilities already built in - without looking at the code,
I am wildly guessing that with some refactoring this logic could maybe
be reused, so that a new headline iteration API would support existing
mechanisms such as tag searches, org-agenda-skip, and sorting?  If so,
IMHO that would be really amazingly powerful :-)

Would be very interested in everyone's thoughts.  Thanks!
Adam


_______________________________________________
Emacs-orgmode mailing list
Remember: use `Reply All' to send replies to the list.
Emacs-orgmode@gnu.org
http://lists.gnu.org/mailman/listinfo/emacs-orgmode


[-- Attachment #1.2: Type: text/html, Size: 4260 bytes --]

[-- Attachment #2: Type: text/plain, Size: 204 bytes --]

_______________________________________________
Emacs-orgmode mailing list
Remember: use `Reply All' to send replies to the list.
Emacs-orgmode@gnu.org
http://lists.gnu.org/mailman/listinfo/emacs-orgmode

Re: FR: headline iteration API

[Adam Spiers]

On Wed, Jun 11, 2008 at 08:57:39AM +0200, Dominik, C. wrote:
> Hi Adam and others,
> 
> I do like the idea of an API to iterate of entries and outline trees.
> For now, I am following this discussion to see what ideas pop up.
> When I find the time, something will be implemented.

Great.  As you can see from my other post, another use case just
popped up.

Am I right in thinking that you must already have a lot of the code
for this?  Presumably agenda generation and export must both do
headline iteration in a similar manner?

Full-blown reporting would be seriously cool.  Think: pretty coloured
graphs showing how the contents of your TODO lists vary over the days,
weeks, months ... :-)  Very useful for trending, planning, ensuring
that the various areas of your life are kept in balance according to
your "30,000-50,000 foot views", as David Allen calls them.

Re: FR: headline iteration API

[Eddward DeVilla]

Are the functions behind C-c C-N, C-c C-p, C-c C-f, C-c C-b & C-c C-u
available?  Seems you could add a function for going to the first
child.  As long as that, C-c C-f & C-c C-b all return something to let
you know there isn't a next, this should be pretty complete.

I guess all you would need would be the following functions which
would move to the correct place or return something to say there isn't
a next/parent/child/sibling etc to signal the end of iteration.

 - doc traversal
    - first-item
      Go to the first item in the file.
    - current-item
      Go to the beginning of the item containing the cursor.
    - next-item
      Go to the item after the current one.
    - previous-item
      Go to item before the current one

 - tree traversal
    - parent-item
      Go to the parent item of the current item
    - first-child-item
      Go to the first item contained in the current item
    - next-sibling-item
      Go to the next item that has the same parent
    - previous-sibling-item
      Go to the previous item that has the same parent

This ought to be enough to try to implement anything else on top of
it.  Did I miss something?

Edd


On Wed, Jun 11, 2008 at 6:06 AM, Adam Spiers <orgmode@adamspiers.org> wrote:
> On Wed, Jun 11, 2008 at 08:57:39AM +0200, Dominik, C. wrote:
>> Hi Adam and others,
>>
>> I do like the idea of an API to iterate of entries and outline trees.
>> For now, I am following this discussion to see what ideas pop up.
>> When I find the time, something will be implemented.
>
> Great.  As you can see from my other post, another use case just
> popped up.
>
> Am I right in thinking that you must already have a lot of the code
> for this?  Presumably agenda generation and export must both do
> headline iteration in a similar manner?
>
> Full-blown reporting would be seriously cool.  Think: pretty coloured
> graphs showing how the contents of your TODO lists vary over the days,
> weeks, months ... :-)  Very useful for trending, planning, ensuring
> that the various areas of your life are kept in balance according to
> your "30,000-50,000 foot views", as David Allen calls them.
>
>
> _______________________________________________
> Emacs-orgmode mailing list
> Remember: use `Reply All' to send replies to the list.
> Emacs-orgmode@gnu.org
> http://lists.gnu.org/mailman/listinfo/emacs-orgmode
>

Re: FR: headline iteration API

[Eddward DeVilla]

On Wed, Jun 11, 2008 at 12:15 PM, Eddward DeVilla <eddward@gmail.com> wrote:
> Are the functions behind C-c C-N, C-c C-p, C-c C-f, C-c C-b & C-c C-u
> available?  Seems you could add a function for going to the first
> child.  As long as that, C-c C-f & C-c C-b all return something to let
> you know there isn't a next, this should be pretty complete.
>
> I guess all you would need would be the following functions which
> would move to the correct place or return something to say there isn't
> a next/parent/child/sibling etc to signal the end of iteration.
>
>  - doc traversal
>    - first-item
>      Go to the first item in the file.
>    - current-item
>      Go to the beginning of the item containing the cursor.
>    - next-item
>      Go to the item after the current one.
>    - previous-item
>      Go to item before the current one
>
>  - tree traversal
>    - parent-item
>      Go to the parent item of the current item
>    - first-child-item
>      Go to the first item contained in the current item
>    - next-sibling-item
>      Go to the next item that has the same parent
>    - previous-sibling-item
>      Go to the previous item that has the same parent
>
> This ought to be enough to try to implement anything else on top of
> it.  Did I miss something?

Just to respond to myself, I did miss something.  If we want a
convenient base api, we probably ought to have an equivalent to the
doc traversal functions for iterating through all the items in the
agenda files.  I'm not sure what the correct behavior should be in
that case if you start in a file that is not in the list of agenda
files.  I was precise about the sibling functions to handle the
following if you call next-sibling-item from meanie.

* eenie
*** meanie
** minie
* Hey Moe!

Edd

Re: FR: headline iteration API

[Carsten Dominik]

On Jun 10, 2008, at 12:28 PM, Adam Spiers wrote:
> Yes.  There will be many intricacies involved in iteration, and
> probably only Carsten knows them all - without an API we would
> individually be reinventing the same wheel over and over ... badly.


Hi Adam, Daniel, and others,

the mapping API is in place and already quite general.
It supports different scopes like the current tree, or the file,
or all agenda files, or even all agenda files with associated
archives.  You can specify a tags/property/todo match to
restrict the mapping to selected entries.  And you can use the full
org-agenda-skip mechanism (but you might want to do these checks in
the mapping function, of course).

Adam, as you suspected, this is only a small bit of code, for the
rest it re-uses the agenda stuff.

Documentation is in Appendix B6 of the manual.

I am very curious to see what kind of tricks you will do with this,
to keep us posted!

- Carsten



>
>
>
> _______________________________________________
> Emacs-orgmode mailing list
> Remember: use `Reply All' to send replies to the list.
> Emacs-orgmode@gnu.org
> http://lists.gnu.org/mailman/listinfo/emacs-orgmode

Re: FR: headline iteration API

[Adam Spiers]

On Thu, Jun 12, 2008 at 08:46:47AM +0200, Carsten Dominik wrote:
> 
> On Jun 10, 2008, at 12:28 PM, Adam Spiers wrote:
> >Yes.  There will be many intricacies involved in iteration, and
> >probably only Carsten knows them all - without an API we would
> >individually be reinventing the same wheel over and over ... badly.
> 
> 
> Hi Adam, Daniel, and others,
> 
> the mapping API is in place and already quite general.
> It supports different scopes like the current tree, or the file,
> or all agenda files, or even all agenda files with associated
> archives.  You can specify a tags/property/todo match to
> restrict the mapping to selected entries.  And you can use the full
> org-agenda-skip mechanism (but you might want to do these checks in
> the mapping function, of course).
> 
> Adam, as you suspected, this is only a small bit of code, for the
> rest it re-uses the agenda stuff.
> 
> Documentation is in Appendix B6 of the manual.

Ooo, shiny! :-)

> I am very curious to see what kind of tricks you will do with this,
> to keep us posted!

Sure - but first I need to surmount this compilation error with the
latest git master:

--------- 8< --------- 8< --------- 8< --------- 8< --------- 8< ---------
$ make
Loading /usr/share/emacs/site-lisp/site-start.d/vm-mode-init.el (source)...

In end of data:
org-compat.el:254:1:Warning: the function `add-local-hook' is not known to be
    defined.
Wrote /home/adam/lib/emacs/major-modes/org-mode.git/lisp/org-compat.elc
emacs -batch -q -eval "(progn (add-to-list (quote load-path) (expand-file-name \"./lisp/\")) (add-to-list (quote load-path) \"/usr/local/share/emacs/site-lisp\"))" -f batch-byte-compile lisp/org-faces.el
Loading /usr/share/emacs/site-lisp/site-start.d/focus-init.el (source)...
Loading /usr/share/emacs/site-lisp/site-start.d/gnuplot-init.el (source)...
Loading /usr/share/emacs/site-lisp/site-start.d/gtypist-init.el (source)...
Loading /usr/share/emacs/site-lisp/site-start.d/igrep-init.el (source)...
Loading /usr/share/emacs/site-lisp/site-start.d/lilypond-init.el (source)...
Loading /usr/share/emacs/site-lisp/site-start.d/muse-init.el (source)...
Loading /usr/share/emacs/site-lisp/site-start.d/php-mode-init.el (source)...
Loading /usr/share/emacs/site-lisp/site-start.d/po-mode-init.el (source)...
Loading /usr/share/emacs/site-lisp/site-start.d/psgml-init.el (source)...
Loading /usr/share/emacs/site-lisp/site-start.d/rpm-spec-mode-init.el (source)...
Loading /usr/share/emacs/site-lisp/site-start.d/rpmdev-init.el (source)...
Loading /usr/share/emacs/site-lisp/site-start.d/ruby-mode-init.el (source)...
Loading /usr/share/emacs/site-lisp/site-start.d/vm-mode-init.el (source)...
Wrote /home/adam/lib/emacs/major-modes/org-mode.git/lisp/org-faces.elc
emacs -batch -q -eval "(progn (add-to-list (quote load-path) (expand-file-name \"./lisp/\")) (add-to-list (quote load-path) \"/usr/local/share/emacs/site-lisp\"))" -f batch-byte-compile lisp/org.el
Loading /usr/share/emacs/site-lisp/site-start.d/focus-init.el (source)...
Loading /usr/share/emacs/site-lisp/site-start.d/gnuplot-init.el (source)...
Loading /usr/share/emacs/site-lisp/site-start.d/gtypist-init.el (source)...
Loading /usr/share/emacs/site-lisp/site-start.d/igrep-init.el (source)...
Loading /usr/share/emacs/site-lisp/site-start.d/lilypond-init.el (source)...
Loading /usr/share/emacs/site-lisp/site-start.d/muse-init.el (source)...
Loading /usr/share/emacs/site-lisp/site-start.d/php-mode-init.el (source)...
Loading /usr/share/emacs/site-lisp/site-start.d/po-mode-init.el (source)...
Loading /usr/share/emacs/site-lisp/site-start.d/psgml-init.el (source)...
Loading /usr/share/emacs/site-lisp/site-start.d/rpm-spec-mode-init.el (source)...
Loading /usr/share/emacs/site-lisp/site-start.d/rpmdev-init.el (source)...
Loading /usr/share/emacs/site-lisp/site-start.d/ruby-mode-init.el (source)...
Loading /usr/share/emacs/site-lisp/site-start.d/vm-mode-init.el (source)...

In toplevel form:
\30\303EE\207\304\303rrF\207" [place x setq cons callf2] 5 ("/usr/share/emacs/22.1/lisp/emacs-lisp/cl.elc" . 2669)], 1
make: *** [lisp/org.elc] Error 1

$ rpm -qf /usr/share/emacs/22.1/lisp/emacs-lisp/cl.elc
emacs-common-22.1-4.fc7
--------- 8< --------- 8< --------- 8< --------- 8< --------- 8< ---------

How is one supposed to figure out which part of lisp/org.el is causing that?
My elisp-fu isn't strong enough.

Re: FR: headline iteration API

[Carsten Dominik]

Fixed, thanks.

- Carsten

On Jun 12, 2008, at 12:16 PM, Adam Spiers wrote:

>
> Ooo, shiny! :-)
>
>> I am very curious to see what kind of tricks you will do with this,
>> to keep us posted!
>
> Sure - but first I need to surmount this compilation error with the
> latest git master:
>
> --------- 8< --------- 8< --------- 8< --------- 8< --------- 8<  
> ---------
> $ make
> Loading /usr/share/emacs/site-lisp/site-start.d/vm-mode-init.el  
> (source)...
>
> In end of data:
> org-compat.el:254:1:Warning: the function `add-local-hook' is not  
> known to be
>    defined.
> Wrote /home/adam/lib/emacs/major-modes/org-mode.git/lisp/org- 
> compat.elc
> emacs -batch -q -eval "(progn (add-to-list (quote load-path) (expand- 
> file-name \"./lisp/\")) (add-to-list (quote load-path) \"/usr/local/ 
> share/emacs/site-lisp\"))" -f batch-byte-compile lisp/org-faces.el
> Loading /usr/share/emacs/site-lisp/site-start.d/focus-init.el  
> (source)...
> Loading /usr/share/emacs/site-lisp/site-start.d/gnuplot-init.el  
> (source)...
> Loading /usr/share/emacs/site-lisp/site-start.d/gtypist-init.el  
> (source)...
> Loading /usr/share/emacs/site-lisp/site-start.d/igrep-init.el  
> (source)...
> Loading /usr/share/emacs/site-lisp/site-start.d/lilypond-init.el  
> (source)...
> Loading /usr/share/emacs/site-lisp/site-start.d/muse-init.el  
> (source)...
> Loading /usr/share/emacs/site-lisp/site-start.d/php-mode-init.el  
> (source)...
> Loading /usr/share/emacs/site-lisp/site-start.d/po-mode-init.el  
> (source)...
> Loading /usr/share/emacs/site-lisp/site-start.d/psgml-init.el  
> (source)...
> Loading /usr/share/emacs/site-lisp/site-start.d/rpm-spec-mode- 
> init.el (source)...
> Loading /usr/share/emacs/site-lisp/site-start.d/rpmdev-init.el  
> (source)...
> Loading /usr/share/emacs/site-lisp/site-start.d/ruby-mode-init.el  
> (source)...
> Loading /usr/share/emacs/site-lisp/site-start.d/vm-mode-init.el  
> (source)...
> Wrote /home/adam/lib/emacs/major-modes/org-mode.git/lisp/org-faces.elc
> emacs -batch -q -eval "(progn (add-to-list (quote load-path) (expand- 
> file-name \"./lisp/\")) (add-to-list (quote load-path) \"/usr/local/ 
> share/emacs/site-lisp\"))" -f batch-byte-compile lisp/org.el
> Loading /usr/share/emacs/site-lisp/site-start.d/focus-init.el  
> (source)...
> Loading /usr/share/emacs/site-lisp/site-start.d/gnuplot-init.el  
> (source)...
> Loading /usr/share/emacs/site-lisp/site-start.d/gtypist-init.el  
> (source)...
> Loading /usr/share/emacs/site-lisp/site-start.d/igrep-init.el  
> (source)...
> Loading /usr/share/emacs/site-lisp/site-start.d/lilypond-init.el  
> (source)...
> Loading /usr/share/emacs/site-lisp/site-start.d/muse-init.el  
> (source)...
> Loading /usr/share/emacs/site-lisp/site-start.d/php-mode-init.el  
> (source)...
> Loading /usr/share/emacs/site-lisp/site-start.d/po-mode-init.el  
> (source)...
> Loading /usr/share/emacs/site-lisp/site-start.d/psgml-init.el  
> (source)...
> Loading /usr/share/emacs/site-lisp/site-start.d/rpm-spec-mode- 
> init.el (source)...
> Loading /usr/share/emacs/site-lisp/site-start.d/rpmdev-init.el  
> (source)...
> Loading /usr/share/emacs/site-lisp/site-start.d/ruby-mode-init.el  
> (source)...
> Loading /usr/share/emacs/site-lisp/site-start.d/vm-mode-init.el  
> (source)...
>
> In toplevel form:
> \30\303EE\207\304\303rrF\207" [place x setq cons callf2] 5 ("/usr/ 
> share/emacs/22.1/lisp/emacs-lisp/cl.elc" . 2669)], 1
> make: *** [lisp/org.elc] Error 1

>
>
> $ rpm -qf /usr/share/emacs/22.1/lisp/emacs-lisp/cl.elc
> emacs-common-22.1-4.fc7
> --------- 8< --------- 8< --------- 8< --------- 8< --------- 8<  
> ---------
>
> How is one supposed to figure out which part of lisp/org.el is  
> causing that?
> My elisp-fu isn't strong enough.

Yes, this is hard to find.  I just happend to remember that I did  
something dangerous with "push".

- Carsten

Re: FR: headline iteration API

[Max Mikhanosha]

At Wed, 11 Jun 2008 12:15:11 -0500,
Eddward DeVilla wrote:

>  - doc traversal
>     - first-item
>       Go to the first item in the file.
>     - current-item
>       Go to the beginning of the item containing the cursor.
>     - next-item
>       Go to the item after the current one.
>     - previous-item
>       Go to item before the current one

Some time ago I wrote the following for myself, and I find that I use
these bindings several times a day.

(defun my-org-skip-forward (arg)
  "Move forward to the next visible 2nd or greater level heading,
skipping headings of the same level as the starting position"
  (interactive "p")
  (let ((initial-level (org-outline-level))
        (done nil))
    (while (not done)
      (outline-next-visible-heading arg)
      (let ((level (org-outline-level)))
        (when (and (> initial-level 1) (= level 1))
           (setq initial-level -1))
        (setq done (or
                    (and (< arg 0) (bobp)) 
                    (and (> arg 0) (eobp))
                    (and (not (= level initial-level)))))))))

(defun my-org-skip-backward (arg)
  (interactive "p")
  (my-org-skip-forward (- arg)))

The idea is that point is moved to the next visible boundary where
outline level changes. This is useful when you have your file in
collapsed state with only a few trees expanded. Above command allows
one to quickly jump between such trees.

Also useful for navigating result of a sparse tree search in the same
manner.

Regards,
  Max