A new module for Org-mode: Org-X

A new module for Org-mode: Org-X

[John Wiegley]

Hello all,

I've been working a new set of modules for Org to make it easy to associate
Org entries with data in external systems, such as Redmine, Bugzilla,
WordPress, or even your e-mail Inbox.  It's called Org-X, as its meant to
simplify writing linkup code for any system X.

It has two parts: one for users, and one for Org programmers.  The user code
will:

  - create a new entry on the selected remote based on an Org subtree
  - merge changes from all remotes into a subtree
  - push changes from the local subtree up to the remote(s)

For example, I currently use this code with Redmine to simplify creating new
bugs from existing Org tasks, and to push and pull any new comments.

For Org programmers, Org-X provides a data abstraction to ease working with
Org data in a programmatic way.  To add a tag to the current Org entry with
Org-X looks like this right now:

  (let ((entry (org-x-parse-entry)))
    (org-x-add-tag entry "Hello")
    (org-x-replace-entry entry))

Where the value comes in for programmers is that you can "propagate" such
changes to all associated external systems by just passing "t" for the
propagate parameter:

  (let ((entry (org-x-parse-entry)))
    (org-x-add-tag entry "Hello" t))

Without having to know what those remote systems even are.  Of course, Org
itself is a backend, which is why the call to `org-x-replace-entry' is no
longer needed here, since propagation will achieve the same affect.

The set of applicable backends is determined by context, and the appropriate
actions decided upon by each backend who "answers" for a given Org entry.

Anyway, the code is still in the beginnings stages, and in rough shape, with
the API undergoing fluctuations on a daily basis.  But I thought I'd let
people know, as maybe some would like to help flesh out the basics while it's
still so unformed.  My org-x branch is here:

    git clone git://github.com/jwiegley/org-mode.git
    git checkout -t origin/org-x
or
    git remote add -f jwiegley git://github.com/jwiegley/org-mode.git
    git checkout -t jwiegley/org-x

I'm currently working on backends for Redmine, Bugzilla and WordPress, since I
have an immediate need for them; but I also have ideas for others, such as a
backend that creates back-linked Org tasks for every "FIXME" comment in a
source code project...

Thanks, John

Re: A new module for Org-mode: Org-X

[Karl Voit]

* John Wiegley <jwiegley@gmail.com> wrote:
> Hello all,

Hi!

> I've been working a new set of modules for Org to make it easy to associate
> Org entries with data in external systems, such as Redmine, Bugzilla,
> WordPress, or even your e-mail Inbox.  It's called Org-X, as its meant to
> simplify writing linkup code for any system X.

Wow. This looks like exactly what I want to achieve with the
«Memacs» project on [1]. We certainly need to exchange ideas and
code :-)

> It has two parts: one for users, and one for Org programmers.  

In our project we do have the very same thing: the user-part is
called «Nexus» and the central part is called «Memacs».

I already proposed my ideas here with my posting «Represent
*everything* in Org-mode»[2].

> The user code
> will:
>
>   - create a new entry on the selected remote based on an Org subtree
>   - merge changes from all remotes into a subtree
>   - push changes from the local subtree up to the remote(s)
>
> For example, I currently use this code with Redmine to simplify creating new
> bugs from existing Org tasks, and to push and pull any new comments.
>
> For Org programmers, Org-X provides a data abstraction to ease working with
> Org data in a programmatic way.  To add a tag to the current Org entry with
> Org-X looks like this right now:
>
>   (let ((entry (org-x-parse-entry)))
>     (org-x-add-tag entry "Hello")
>     (org-x-replace-entry entry))
>
> Where the value comes in for programmers is that you can "propagate" such
> changes to all associated external systems by just passing "t" for the
> propagate parameter:
>
>   (let ((entry (org-x-parse-entry)))
>     (org-x-add-tag entry "Hello" t))

Our plan is to use Python to simply generate org (archive) files
from different sources since we do have no ELISP knowledge and
Python does already have beautiful libraries for RegEx, string
manipulation, XML, JSON, ...

> Without having to know what those remote systems even are.  Of course, Org
> itself is a backend, which is why the call to `org-x-replace-entry' is no
> longer needed here, since propagation will achieve the same affect.

Our plan includes Nexus that re-generate their org-file-content on
each invocation OR append new events at the end.

> The set of applicable backends is determined by context, and the appropriate
> actions decided upon by each backend who "answers" for a given Org entry.

Sounds cool :-)

> Anyway, the code is still in the beginnings stages, and in rough shape, with
> the API undergoing fluctuations on a daily basis.  But I thought I'd let
> people know, as maybe some would like to help flesh out the basics while it's
> still so unformed.  

We do not have any code yet but two of my students began to work on
the project.

> My org-x branch is here:
>
>     git clone git://github.com/jwiegley/org-mode.git
>     git checkout -t origin/org-x
> or
>     git remote add -f jwiegley git://github.com/jwiegley/org-mode.git
>     git checkout -t jwiegley/org-x
>
> I'm currently working on backends for Redmine, Bugzilla and WordPress, since I
> have an immediate need for them; but I also have ideas for others, such as a
> backend that creates back-linked Org tasks for every "FIXME" comment in a
> source code project...

We do have lots of ideas for other «Nexus» or «backends» how you
call them :-)

  1. https://github.com/novoid/Memacs

  2. Message-ID: <2011-07-18T01-13-54@devnull.Karl-Voit.at>
  Mon, 18 Jul 2011 01:54:08 +0200
  Sorry, gmane does not let me locate using the search feature :-(

-- 
Karl Voit

Re: A new module for Org-mode: Org-X

[John Wiegley]

Karl Voit <devnull@Karl-Voit.at> writes:

> Wow. This looks like exactly what I want to achieve with the
> «Memacs» project on [1]. We certainly need to exchange ideas and
> code :-)

They sure do sound similar in spirit!!  It's like we are tuned to the same
wavelength. :)

> Our plan is to use Python to simply generate org (archive) files from
> different sources since we do have no ELISP knowledge and Python does
> already have beautiful libraries for RegEx, string manipulation, XML, JSON,

I have to say that one of my personal requirements is that such a system be
implemented as much in Emacs Lisp as possible.  I hate debugging code that's
"in between" Emacs and some other system.  Plus, I have excellent
documentation, debugging and profiling resources available to me in Emacs, but
I've never found anything close to equivalent in Python (and I've tried them
all: ipython, pydb, ropemacs, etc).  Developing in Emacs is just so efficient
and fun, that I do it as much for recreation as to get things done.

> We do not have any code yet but two of my students began to work on the
> project.

Are they Lisp programmers?  If so, perhaps we could combine our efforts.
There are many backends I'd like to write -- such as one that automatically
records "ticked" articles in my Inbox as linked Org tasks -- but I only have
so much time in each day.

Since my own needs are immediate for this project, I should have the first
three backends for Org-X completed by month's end.  By that time, the core API
should be stable and I'll be ready to merge this code into main.  I've also
been writing for org.texi at the same time, so it will come fully documented
when it arrives.

John

Re: A new module for Org-mode: Org-X

[Karl Voit]

* John Wiegley <jwiegley@gmail.com> wrote:
> Karl Voit <devnull@Karl-Voit.at> writes:
>
>> Wow. This looks like exactly what I want to achieve with the
>> «Memacs» project on [1]. We certainly need to exchange ideas and
>> code :-)
>
> They sure do sound similar in spirit!!  It's like we are tuned to the same
> wavelength. :)

*g*

>> Our plan is to use Python to simply generate org (archive) files from
>> different sources since we do have no ELISP knowledge and Python does
>> already have beautiful libraries for RegEx, string manipulation, XML, JSON,
>
> I have to say that one of my personal requirements is that such a system be
> implemented as much in Emacs Lisp as possible.  I hate debugging code that's
> "in between" Emacs and some other system.  Plus, I have excellent
> documentation, debugging and profiling resources available to me in Emacs, but
> I've never found anything close to equivalent in Python (and I've tried them
> all: ipython, pydb, ropemacs, etc).  Developing in Emacs is just so efficient
> and fun, that I do it as much for recreation as to get things done.

I am sure that ELISP is cool. But here, we do not have any LISP
knowledge at all and I hardly know anyone in our hacking community
who is really into ELISP I am afraid. So I have to stick to my
(local) resources.

In my point of view, it is no problem at all because the «API» we do
have is the format of Org-mode files. Memacs is - unlike your system
- not done with Emacs at all: Python scripts generate Org-mode files
and that is all.

This is a fundamental difference in our attempts - I know.

Maybe our systems could cooperate like Memacs doing
in-between-filtering and Org-X the integration to Org-mode or
similar.

>> We do not have any code yet but two of my students began to work on the
>> project.
>
> Are they Lisp programmers?  

No. I know not a single ELISP programmer here at my side :-(

From my point of view - and I am sure this may differ from reality!
- ELISP is some kind of niche language that is slowly going down in
popularity. Again: this is not what I want it to be, this is my very
personal experience.

> If so, perhaps we could combine our efforts.

Sure. But we can not provide ELISP code. So my current plan is
still: stick to Python, let the finished (auto-generated) Org-mode
files be the «API». It's better to do *something* (in Python) than
to do nothing at all.

> Since my own needs are immediate for this project, I should have the first
> three backends for Org-X completed by month's end.  By that time, the core API
> should be stable and I'll be ready to merge this code into main.  I've also
> been writing for org.texi at the same time, so it will come fully documented
> when it arrives.

I sure will take a look at it!

We do not have any plan when first Nexus and Memacs will be out. We
are still in the early planning phase.

-- 
Karl Voit

Re: A new module for Org-mode: Org-X

[John Wiegley]

Karl Voit <devnull@Karl-Voit.at> writes:

> In my point of view, it is no problem at all because the «API» we do have is
> the format of Org-mode files. Memacs is - unlike your system - not done with
> Emacs at all: Python scripts generate Org-mode files and that is all.

Very likely Org-X will be able to make sure of your backends in some way, to
provide functionality for backends that don't get written Emacs Lisp.

John

Re: A new module for Org-mode: Org-X

[Sebastien Vauban]

Hi John,

John Wiegley wrote:
> I have to say that one of my personal requirements is that such a system be
> implemented as much in Emacs Lisp as possible.  I hate debugging code that's
> "in between" Emacs and some other system.  Plus, I have excellent
> documentation, debugging and profiling resources available to me in Emacs, but
> I've never found anything close to equivalent in Python (and I've tried them
> all: ipython, pydb, ropemacs, etc).  Developing in Emacs is just so efficient
> and fun, that I do it as much for recreation as to get things done.

Would you have a .emacs part about this (debugging, profiling, etc.) that
you're willing to share. I am confident that I could jump easier into full
Emacs Lisp if I would have a better environment.

Thanks...

Best regards,
  Seb

-- 
Sebastien Vauban

Re: A new module for Org-mode: Org-X

[John Wiegley]

[-- Attachment #1: Type: text/plain, Size: 590 bytes --]

"Sebastien Vauban" <wxhgmqzgwmuf-geNee64TY+gS+FvcfC7Uqw@public.gmane.org>
writes:

> Would you have a .emacs part about this (debugging, profiling, etc.) that
> you're willing to share. I am confident that I could jump easier into full
> Emacs Lisp if I would have a better environment.

I'm attaching my lang-emacs-lisp.el file.  The really big modules to know are:

  - edebug  <-- sheer brilliance
  - eldoc
  - elint
  - elp (the Emacs Profiler)

If you do a LOT of code hacking in Lisp, then paredit and redshank are
invaluable.  If you only dabble/debug, they are unnecessary.

John


[-- Attachment #2: lang-emacs-lisp.el --]
[-- Type: application/emacs-lisp, Size: 2347 bytes --]

Re: A new module for Org-mode: Org-X

[Sebastien Vauban]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset=us-ascii, Size: 3167 bytes --]

Hi John,

John Wiegley wrote:
> "Sebastien Vauban" <wxhgmqzgwmuf-geNee64TY+gS+FvcfC7Uqw@public.gmane.org>
> writes:
>
>> Would you have a .emacs part about this (debugging, profiling, etc.) that
>> you're willing to share. I am confident that I could jump easier into full
>> Emacs Lisp if I would have a better environment.
>
> I'm attaching my lang-emacs-lisp.el file.  The really big modules to know are:
>
>   - edebug  <-- sheer brilliance
>   - eldoc
>   - elint
>   - elp (the Emacs Profiler)
>
> If you do a LOT of code hacking in Lisp, then paredit and redshank are
> invaluable.  If you only dabble/debug, they are unnecessary.
>
> John
>
> ;;;_ * eldoc
>
> (add-hook 'emacs-lisp-mode-hook (lambda () (require 'edebug)))
>
> ;;;_ * eldoc
>
> (eval-after-load "eldoc"
>   '(diminish 'eldoc-mode))
>
> ;;;_ * elint
>
> (defun elint-current-buffer ()
>   (interactive)
>   (elint-initialize)
>   (elint-current-buffer))
>
> (eval-after-load "elint"
>   '(progn
>      (add-to-list 'elint-standard-variables 'current-prefix-arg)
>      (add-to-list 'elint-standard-variables 'command-line-args-left)
>      (add-to-list 'elint-standard-variables 'buffer-file-coding-system)
>      (add-to-list 'elint-standard-variables 'emacs-major-version)
>      (add-to-list 'elint-standard-variables 'window-system)))
>
> ;;;_ * emacs-lisp
>
> (add-hook 'emacs-lisp-mode-hook 'turn-on-auto-fill)
>
> (font-lock-add-keywords 'emacs-lisp-mode
>                         '(("(\\(lambda\\)\\>"
>                            (0 (ignore
>                                (compose-region (match-beginning 1)
>                                                (match-end 1) ?λ))))))
>
> (defun elisp-indent-or-complete (&optional arg)
>   (interactive "p")
>   (call-interactively 'lisp-indent-line)
>   (unless (or (looking-back "^\\s-*")
> 	      (bolp)
> 	      (not (looking-back "[-A-Za-z0-9_*+/=<>!?]+")))
>     (call-interactively 'lisp-complete-symbol)))
>
> (eval-after-load "lisp-mode"
>   '(define-key emacs-lisp-mode-map [tab] 'elisp-indent-or-complete))
>
> ;;;_  + column-marker
>
> (add-hook 'emacs-lisp-mode-hook (lambda () (column-marker-1 79)))
>
> ;;;_  + highlight-parentheses
>
> (autoload 'highlight-parentheses-mode "highlight-parentheses")
>
> (add-hook 'emacs-lisp-mode-hook 'highlight-parentheses-mode)
>
> (eval-after-load "highlight-parentheses"
>   '(diminish 'highlight-parentheses-mode))
>
> ;;;_  + paredit
>
> (autoload 'paredit-mode "paredit"
>   "Minor mode for pseudo-structurally editing Lisp code." t)
> (autoload 'turn-on-paredit-mode "paredit"
>   "Minor mode for pseudo-structurally editing Lisp code." t)
>
> (add-hook 'emacs-lisp-mode-hook 'turn-on-paredit-mode)
>
> (eval-after-load "paredit"
>   '(diminish 'paredit-mode))
>
> ;;;_  + redshank
>
> (autoload 'redshank-mode "redshank"
>   "Minor mode for restructuring Lisp code (i.e., refactoring)." t)
>
> (add-hook 'emacs-lisp-mode-hook #'(lambda () (redshank-mode +1)))
>
> (eval-after-load "redshank"
>   '(diminish 'redshank-mode))
>
> ;;; lang-emacs-lisp.el ends here

Thank you so much for this info. I will have a deep look at every line!

Best regards,
  Seb

-- 
Sebastien Vauban

Re: A new module for Org-mode: Org-X

[Achim Gratz]

John Wiegley <jwiegley@gmail.com> writes:
> I've been working a new set of modules for Org to make it easy to associate
> Org entries with data in external systems, such as Redmine, Bugzilla,
> WordPress, or even your e-mail Inbox.  It's called Org-X, as its meant to
> simplify writing linkup code for any system X.

You're on to something... :-)

I can easily see how this works with bug/issue trackers.  What I'm not
so sure is how you imagine it to work with files.  For my part I'm
hoping that it might be able to get some interface to the various bits
and pieces I've bookmarked and sometimes put into local versions with
the ScrapBook Firefox extension.

A few random comments:

I'd check source code for FIXME, not TODO.

If org-x-states is backend specific, why not let the backend define it?

LZW: is there really nothing in Emacs that already does this? In any
case it seems to belong into an own library and should probably be
unbundled from org before long.


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

SD adaptations for KORG EX-800 and Poly-800MkII V0.9:
http://Synth.Stromeko.net/Downloads.html#KorgSDada

Re: A new module for Org-mode: Org-X

[John Wiegley]

Achim Gratz <Stromeko@nexgo.ed> writes:

> I can easily see how this works with bug/issue trackers.  What I'm not so
> sure is how you imagine it to work with files.  For my part I'm hoping that
> it might be able to get some interface to the various bits and pieces I've
> bookmarked and sometimes put into local versions with the ScrapBook Firefox
> extension.

The backend gets to define what the parts of an entry are.  Some backends will
import actual content into the related Org entry, while others will only
import a link back to the original content.  This is what I would do for a
backend that monitors files within an "Inbox" directory, for example.

> I'd check source code for FIXME, not TODO.

Sure, this would be configurable through both a customization variable (for
setting default behavior), and through an inherited property (as part of the
backend context).

> If org-x-states is backend specific, why not let the backend define it?

It is defined, by the ox-org backend. :)  I imagine that very soon now, I'll
pay attention to SEQ_TODO and the usual Org variable, rather than having my
own custom variable for this.

> LZW: is there really nothing in Emacs that already does this? In any case it
> seems to belong into an own library and should probably be unbundled from
> org before long.

There's really not!  I wrote it because I imagine at some point wanting to
take a snapshot of the "state" of an entry, for the purpose of accelerating
certain backends, but I haven't yet found that use.  For now, it's just living
in ox-lzw so that it's under version control.  If it becomes useful, I'll
submit it to Emacs separately and ahead of Org-X.

John

Re: A new module for Org-mode: Org-X

[Achim Gratz]

John Wiegley <jwiegley@gmail.com> writes:
> The backend gets to define what the parts of an entry are.  Some backends will
> import actual content into the related Org entry, while others will only
> import a link back to the original content.  This is what I would do for a
> backend that monitors files within an "Inbox" directory, for example.

Still clear as mud... I'll have to give this more time to think over.

>> LZW: is there really nothing in Emacs that already does this? In any case it
>> seems to belong into an own library and should probably be unbundled from
>> org before long.
>
> There's really not!  I wrote it because I imagine at some point wanting to
> take a snapshot of the "state" of an entry, for the purpose of accelerating
> certain backends, but I haven't yet found that use.  For now, it's just living
> in ox-lzw so that it's under version control.  If it becomes useful, I'll
> submit it to Emacs separately and ahead of Org-X.

That's what I was suspecting.  It appears you're re-inventing certain
aspects of serialization/deserialization and that might come in handy
for a lot more things.  For starters, if it was built-in into Emacs that
whole business of dumping a bootstrapped Emacs and doing horrible things
to the resulting file to be able to get it back to life would probably
become uneccesary...


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

Factory and User Sound Singles for Waldorf Q+, Q and microQ:
http://Synth.Stromeko.net/Downloads.html#WaldorfSounds