Feature Request - Active and inactive links.

Feature Request - Active and inactive links.

[Tim O'Callaghan]

This is something i have been pondering for a while, and after
seeing the recent discussion on links from org files, thought i
would put forward for discussion. It is an attempt to kill two
birds with one stone. The first is collaboration, the second is a
selective trigger mechanism for generating Agenda items.

Currently Org mode supports links like:
     http://www.astro.uva.nl/~dominik
     file:/home/dominik/images/jupiter.jpg
     news:comp.emacs
     etc....

I would like to propose the concept of 'Active' links, based on
the above. The idea being that some links are marked such that
when Org is building an agenda, it includes these links as if they
were in the org-agenda-files list.

An active link could be prefixed by a + sign, possibly with
embedded metainformation for the agenda.

Some possible examples:
-  read/write remote org file for collaboration via efs/angeftp
 +file:/me@my-home-machine.com:/home/me/personal.org
- read only remote org file for collaboration in category work
 +work+http://www.astro.uva.nl/~dominik/remote.org
- read only remote ical file of local whats-on information.
 +whatson+ical:http://upcoming.yahoo.com/calendar/v2/place/upI5ACueA5szd_8-

Some thought would need to be put into how agenda handles read
only links. In a collaborating read only situation, a masking or
supplementing mechanism might be needed.

This mechanism could conceivably be used to auto-build
org-agenda-files by traversing the active links.

What do people think?

Tim.

Re: Feature Request - Active and inactive links.

[Bastien]

Hi Tim,

"Tim O'Callaghan" <tim.ocallaghan@gmail.com> writes:

> Currently Org mode supports links like:
>      http://www.astro.uva.nl/~dominik
>      file:/home/dominik/images/jupiter.jpg
>      news:comp.emacs
>      etc....
>
> I would like to propose the concept of 'Active' links, based on
> the above. The idea being that some links are marked such that
> when Org is building an agenda, it includes these links as if they
> were in the org-agenda-files list.

It's been a while since you posted this message... I think I don't
really understand the core idea here.  Can you elaborate a bit more?
What kind of information do you want to attach to links?  what for?  
in what context should this information be displayed?  processed?

Thanks for providing further details!

-- 
Bastien

Feature Request - Active and inactive links.

[Tim O'Callaghan]

On 10/12/2007, Bastien <bzg@altern.org> wrote:
> Hi Tim,
>
> "Tim O'Callaghan" <tim.ocallaghan@gmail.com> writes:
>
> > Currently Org mode supports links like:
> >      http://www.astro.uva.nl/~dominik
> >      file:/home/dominik/images/jupiter.jpg
> >      news:comp.emacs
> >      etc....
> >
> > I would like to propose the concept of 'Active' links, based on
> > the above. The idea being that some links are marked such that
> > when Org is building an agenda, it includes these links as if they
> > were in the org-agenda-files list.
>
> It's been a while since you posted this message... I think I
> don't really understand the core idea here.  Can you elaborate a
> bit more?  What kind of information do you want to attach to
> links?  what for?  in what context should this information be
> displayed?  processed?
> Thanks for providing further details!
>

Sorry if i was unclear, I thought my examples would it explain most of
it, my bad.

Here it is in more detail. :)

The core idea - Active links - is a link that specifies a
resource (usually remote) that will be included (and possibly
preprocessed) when you compile an agenda. They should look and
act like normal links, but be handled differently when an agenda
is compiled.

Where this idea came from.

I have a hacked together function that i use (see my rusty elisp
below) that creates org-files from ical URLs. I use this to
include my google calendar and other published events in my
agenda.

With the addition of org-add-link-type (described in Appendix
A-2 in the org manual), i could create a link type that would
convert an ical link to an org file on opening. This is great,
but i could not then auto-include that in buffed file in my
agenda without saving it and adding it to org-agenda-files.

What would be needed would be some kind of flag or indicator
that the link should be processed when creating an agenda
buffer. It would need to be assumed that the link is, or will be
preprocessed into, an org file.

This led me to consider the consequences of an 'Active Link' and
how to make it a more general and flexible concept.

* The Agenda is not passive, it modifies its source files.
= This could be mitigated using meta-tagging of read only
resources. Another more finicky method could be a file of
negative or modification assertions that change or remove a read
only link before inclusion in the Agenda. Possibly a table of
specific org node search links that replace the target with the stored org node.

* The Agenda only processes the org-agenda-files list.
= Let org build the list recursively from active links with one
org-file as the head of the tree. This would have the benefit of
letting you build different agendas based on the first org file
referenced. Note - might need to force read only active links to
be leaf nodes (i.e not recurse into them).

* The Active Link referenced is no longer an agenda item.
= If you remove the org file or its link from the within agenda,
you change the Active Link to an inactive link globally. That
is, in all of the linked files in the current org-agenda files
list.

* What if you open an active link from an org-mode buffer?
= Undecided possibly configurable? I would say open the link in
its natural state.

* What use is a an active link to a remote read-only org file?
= Collaboration. I can think of many scenarios, but the one i
like is where my wife can just update a text file or blog post
or whatever to update my agenda.

* How would you represent an active link so it is obvious?
= An active link could be prefixed by a + sign, possibly with
embedded meta information for the agenda.

Some possible examples:
- read/write remote org file for collaboration (efs/angeftp)
 +file:/me@my-home-machine.com:/home/me/personal.org
- read only remote org file for collaboration in category work
 +work+http://www.astro.uva.nl/~dominik/remote.org
- read only remote ical file of local whats-on information.
+whatson+ical:http://upcoming.yahoo.com/calendar/v2/place/upI5ACueA5szd_8-

The +category+ link prefix idea is because + can be part of a URL.

* Where could you go from here?
= The concept could be extended to allow further integration to
other tools using to-org and from-org pre and post
processing. Using my ical hack for an example, it could possibly
be extended to a read/write WEBDAV link. Say for Outlook or
Sunbird integration.


So thats the idea in more detail, hope it clarifies the idea further...

Tim.

-- Google Calendar hack --
(setq google-ical-org-list
 '(
  ; removed personal links, but left a working public ical link.
  ; each ical link consists of:
  ;("ical link"
  ; "ical link download target file"
  ; "org file created - must be in org-agenda-files")
  ("http://upcoming.yahoo.com/calendar/v2/place/upI5ACueA5szd_8-"
   "~/gettingThingsDone/CalendarSync/UpComing.ics"
   "~/gettingThingsDone/CalendarSync/Upcoming.org")))

(defun toc:goggle-to-org ()
  "get a google calendar and convert it into org dates"
  (interactive)
  (with-temp-buffer
    ;; initialise calendar handling
    (let* ((glist google-ical-org-list))
      ;; iterate through list
      (while (setq entry (pop glist))
        (setq google-ical-url (car entry) local-ical-file
              (nth 1 entry) local-date-file (nth 2 entry))
        ;; Delete the diary local files
        (if (file-exists-p local-ical-file) (delete-file local-ical-file))
        (if (file-exists-p local-date-file) (delete-file local-date-file))
        ;; Get ical file
        (w3-download-url google-ical-url (expand-file-name local-ical-file))
        ;; create an empty ical to process
        (if (not (file-exists-p local-ical-file))
            ((set-buffer (find-file local-ical-file))
             (save-buffer local-ical-file)))
        ;; convert to diary without leading &
        (icalendar-import-file local-ical-file local-date-file t)
        ;; create an empty org file if needed
        (if (not (find-buffer-visiting local-date-file))
            ((set-buffer (file-find local-date-file))
             (save-buffer local-date-file)))
        ;; iCalendar leaves the buffers open
        (if (find-buffer-visiting local-date-file)
              (kill-buffer (find-buffer-visiting local-date-file)))
        (if (find-buffer-visiting local-ical-file)
              (kill-buffer (find-buffer-visiting local-ical-file)))
        ))))

Re: Feature Request - Active and inactive links.

[Christian Egli]

"Tim O'Callaghan" <tim.ocallaghan@gmail.com> writes:

> I have a hacked together function that i use (see my rusty elisp
> below) that creates org-files from ical URLs. I use this to
> include my google calendar and other published events in my
> agenda.

Could you not achieve something along your desired lines with using
includes in your diary file? The info on "Fancy Diary Display" has some
information on how to include additional diary files.

Christian

Re: Feature Request - Active and inactive links.

[Bastien]

Hi Tim,

thanks for the details.  I think I got confused because I couldn't
understand what you mean by a link being "processed" when compiling 
the agenda.  Now I understand that it means some kind of inclusion.

So the issue seems twofold:

1. the first issue is about *including* external Org files (or other
   external resources, although I'm not sure to understand what does
   that mean);

2. the second issue is that of *processing* links to external resources
   when using your Org file as a source for other purposes (agenda view,
   export, etc.)

I think both issues are very interesting but should be carefully (and
maybe separately) thought.  

You're speaking about a link that would include the targetted Org file
into the list of agenda files.  Then attaching meta-data to this link,
you would control how the building of the agenda should process the link
(adding category, etc.)

Some example of what we could do:

- a link to an Org file that should be considered part of the master
  file (at any time: agenda view, export, etc.)  This could be a new
  link type like "org:"

  org:~/home/org/header.org

- a link to a file that should be included for specific export:

  #+BEGIN_LaTeX
  org:~/home/org/latex_footer.org
  #+END_LaTeX

  or maybe, if it's not ambigous:

  org:latex:~/home/org/latex_footer.org

- a link to a file that should only be processed in agenda views:

  org:agenda:~/home/org/other-todo.org

- ... and maybe only for a specific agenda view

  org:agenda:n~/home/org/other-todo.org ("n" being the name of the
  command key in org-agenda-custom-commands)

Where you thinking of something like that?

I'm not sure on how to integrate your idea about specifying categories,
and I doubt this is particularily relevant: the links already belong to
entries that will be categorized.

I'm not sure although about your example with iCal.  Do you think it
could fit with the picture above?

Thanks for this neat idea.  I'm sure we're getting somewhere...

-- 
Bastien

Re: Feature Request - Active and inactive links.

[Tim O'Callaghan]

On 11/12/2007, Bastien <bzg@altern.org> wrote:
> Hi Tim,
>
> thanks for the details.  I think I got confused because I couldn't
> understand what you mean by a link being "processed" when compiling
> the agenda.  Now I understand that it means some kind of inclusion.
>
> So the issue seems twofold:
>
> 1. the first issue is about *including* external Org files (or other
>    external resources, although I'm not sure to understand what does
>    that mean);
>

Yes and no. by *including* external resources i was thinking
remote org files, and things (local or remote) that can be
converted into something usable by org Agenda (only an org file
AFAIK). I use an Ical URL as an example because i use google
calendar, but the Ical link could be local. Or it could be a
nag, remind, an outlook export or whatever.

> 2. the second issue is that of *processing* links to external resources
>    when using your Org file as a source for other purposes (agenda view,
>    export, etc.)
>

Yes. My idea was essentially, when i ask org to create an agenda
buffer, it knows to auto-pull and process each all of these
active links, so as to be able to display them in my Agenda.

> I think both issues are very interesting but should be carefully (and
> maybe separately) thought.
>
> You're speaking about a link that would include the targeted Org file
> into the list of agenda files.  Then attaching meta-data to this link,
> you would control how the building of the agenda should process the link
> (adding category, etc.)
>
> Some example of what we could do:
>
> - a link to an Org file that should be considered part of the master
>   file (at any time: agenda view, export, etc.)  This could be a new
>   link type like "org:"
>
>   org:~/home/org/header.org
>

What is a Master file in this context?

> - a link to a file that should be included for specific export:
>
>   #+BEGIN_LaTeX
>   org:~/home/org/latex_footer.org
>   #+END_LaTeX
>
>   or maybe, if it's not ambiguous:
>
>   org:latex:~/home/org/latex_footer.org
>

Not quite. Though i can see how some might have use for the
export specific org file include.

> - a link to a file that should only be processed in agenda views:
>
>   org:agenda:~/home/org/other-todo.org
>
> - ... and maybe only for a specific agenda view
>
>   org:agenda:n~/home/org/other-todo.org ("n" being the name of the
>   command key in org-agenda-custom-commands)
>
> Where you thinking of something like that?
>

Something similar to the last one.

> I'm not sure on how to integrate your idea about specifying categories,
> and I doubt this is particularly relevant: the links already belong to
> entries that will be categorized.
>

Well the category is more like a category override. If you
consider an imported org: link that is not created by you, then
it may have a different flavor. The adding - or probably better
- superseding of category/meta information gives you the
knowledge needed to search for imported todos for example.

> I'm not sure although about your example with iCal.  Do you think it
> could fit with the picture above?
>

After thinking about it, for external resources, it would be
better to specify a new active link type per resource type
"Ical:" for example.  Also the meta data might be better if
specified per link type to or in the processing code.

> Thanks for this neat idea.  I'm sure we're getting somewhere...
>

Not sure how neat it is, but i know I'll use it :)

Tim.

Re: Feature Request - Active and inactive links.

[Bastien]

"Tim O'Callaghan" <tim.ocallaghan@gmail.com> writes:

>> 1. the first issue is about *including* external Org files (or other
>>    external resources, although I'm not sure to understand what does
>>    that mean);
>
> Yes and no. by *including* external resources i was thinking
> remote org files, and things (local or remote) that can be
> converted into something usable by org Agenda (only an org file
> AFAIK). I use an Ical URL as an example because i use google
> calendar, but the Ical link could be local. Or it could be a
> nag, remind, an outlook export or whatever.

I think we should study this idea incrementally.  The first step is to
see how we can "import" an Org file (since every other file types would
be converted into something that Org should be able to compile.)

>> 2. the second issue is that of *processing* links to external resources
>>    when using your Org file as a source for other purposes (agenda view,
>>    export, etc.)
>
> Yes. My idea was essentially, when i ask org to create an agenda
> buffer, it knows to auto-pull and process each all of these
> active links, so as to be able to display them in my Agenda.

(Maybe the word "link" is confusing at some point: a (web)link is
usually something that you go to, not something that sneaks in.  But
let's say that we try to *import* stuff that are reachable trough a 
link to a local/remote resource.)

>> - a link to an Org file that should be considered part of the master
>>   file (at any time: agenda view, export, etc.)  This could be a new
>>   link type like "org:"
>>
>>   org:~/home/org/header.org
>>
>
> What is a Master file in this context?

The master file is the file that contains references to included files.
For exemple:

,----
| #+TITLE: Master file
| 
| org:included-file.org
`----

,----
| #+TITLE: Included file
| 
| Some text.
`----

Caveat: we should perhaps have only one level of inclusion, to avoid
conflicts emerging from circular references...

> Well the category is more like a category override. If you
> consider an imported org: link that is not created by you, then
> it may have a different flavor. The adding - or probably better
> - superseding of category/meta information gives you the
> knowledge needed to search for imported todos for example.

I'm not sure this is useful.  If you import an external todo.org you
just need to be able to refer to the categories it contains, as usual.
You don't need its categories to supersede those of the master file.
But maybe I don't understand you on this.

> After thinking about it, for external resources, it would be
> better to specify a new active link type per resource type
> "Ical:" for example.  Also the meta data might be better if
> specified per link type to or in the processing code.

I think importing external resources from different formats is a bit
overkill.  Importing Org files is enough.  If the user wants to import
iCal files into Org files (as you do), he can do this in a separate
process.

But this topic is huge.  Let's let it rest a bit...

-- 
Bastien

Re: Feature Request - Active and inactive links.

[Christian Egli]

Tim O'Callaghan <tim.ocallaghan <at> gmail.com> writes:

> Yes. My idea was essentially, when i ask org to create an agenda
> buffer, it knows to auto-pull and process each all of these
> active links, so as to be able to display them in my Agenda.

I still don't quite understand why you do not use existing infrastructure such
as the include statement in diary files and the org-agenda-files variable which
lets you define a list files to be included for agenda generation.

Christian

Re: Re: Feature Request - Active and inactive links.

[Bastien]

Christian Egli <christian.egli@novell.com> writes:

> Tim O'Callaghan <tim.ocallaghan <at> gmail.com> writes:
>
>> Yes. My idea was essentially, when i ask org to create an agenda
>> buffer, it knows to auto-pull and process each all of these
>> active links, so as to be able to display them in my Agenda.
>
> I still don't quite understand why you do not use existing infrastructure such
> as the include statement in diary files and the org-agenda-files variable which
> lets you define a list files to be included for agenda generation.

The include statement in the diary file is enough for inclusion of diary
files.  Adding files in org-agenda-files would be enough when we don't
need to have a certain file included depending on the content we're
processing.

I think the rationale behind Tim's idea is: 

- make it possible to *dynamically* process a list of agenda files
  (instead of the static org-agenda-files)

- make it possible to fetch remote/shared resources so that you can
  *collaborate* over the network with Org files.

The horizon is to handle Org files in a more dynamic, modular,
distributed way © ...  But maybe Tim has better/simpler arguments.

-- 
Bastien