Converting from OmniFocus to org-mode

Converting from OmniFocus to org-mode

[Jay McCarthy]

I'm an avid OmniFocus user and I've been thinking about trying org-mode.

The conversion tool on Worg [1] was inadequate because it didn't
capture whether the repetition in tasks and it didn't handle the notes
field properly for me. I've written a new tool that works directly on
the XML database behind OmniFocus. I specifically ignored completed
tasks, whether something was flagged, and durations, because I don't
use those so I couldn't test them. It is written in Racket [2] and
available here:

https://github.com/jeapostrophe/exp/blob/master/of2org.rkt

However, once I converted my 1.6M OmniFocus database, I got a 252K
org-mode file. Opening this file in org-mode is unbearable. For
example, pressing up after opening the document (and having the cursor
at the end of the buffer) takes 5 seconds to select the last item.
Pressing up again takes 6 seconds. Almost every trivial browsing
operation like this takes 5 seconds. (Except expanding the notes
fields once a sub-tree has been revealed. That's instant.) Finally,
the killer. If I select the last top-level heading and press M-up,
Emacs crashes.  A section of the stack trace is at the end of this
email.

I'm happy to send my org-mode file to a developer to try out, but I
don't want to post it online. FWIW, the file is 6738 lines long and
the final heading starts on line 2632, so M-up is moving a significant
number of lines.

I updated to the latest git version of org-mode to see if anything was
faster, but it wasn't.

I find org-mode theoretically appealing, but as it stands, I can't use
it because it isn't scalable enough. Are there any techniques to
getting around this lack of scalability? I'd appreciated any advice.

Best,

Jay

1. https://bitbucket.org/legoscia/of2org/src
2. http://racket-lang.org/

-- The stack trace --

Process:         Emacs [5146]
Path:            /Users/USER/*/Emacs.app/Contents/MacOS/Emacs
Identifier:      org.gnu.Emacs
Version:         Version 23.3 (9.0)
Code Type:       X86-64 (Native)
Parent Process:  launchd [176]

Date/Time:       2011-08-05 09:36:46.341 -0600
OS Version:      Mac OS X 10.7 (11A511)
Report Version:  9

Interval Since Last Report:          201400 sec
Crashes Since Last Report:           5
Per-App Interval Since Last Report:  231997 sec
Per-App Crashes Since Last Report:   2
Anonymous UUID:                      91A4D920-5BEC-4689-B878-21946C6D436B

Crashed Thread:  0  Dispatch queue: com.apple.main-thread

Exception Type:  EXC_BAD_ACCESS (SIGABRT)
Exception Codes: KERN_INVALID_ADDRESS at 0x00007fff5fc00000

VM Regions Near 0x7fff5fc00000:
    Stack                  00007fff5f400000-00007fff5fc00000 [ 8192K]
rw-/rwx SM=COW  thread 0
-->
    __TEXT                 00007fff61e0e000-00007fff61e43000 [  212K]
r-x/rwx SM=COW  /usr/lib/dyld

Application Specific Information:
objc[5146]: garbage collection is OFF
abort() called

Thread 0 Crashed:: Dispatch queue: com.apple.main-thread
0   libsystem_kernel.dylib        	0x00007fff867cd82a __kill + 10
1   org.gnu.Emacs                 	0x00000001000901ae fatal_error_signal + 286
2   libsystem_c.dylib             	0x00007fff84905cfa _sigtramp + 26
3   libsystem_kernel.dylib        	0x00007fff867cdce2 __pthread_kill + 10
4   libsystem_c.dylib             	0x00007fff848b37d2 pthread_kill + 95
5   libsystem_c.dylib             	0x00007fff848a4a7a abort + 143
6   org.gnu.Emacs                 	0x000000010016f31c ns_term_shutdown + 44
7   org.gnu.Emacs                 	0x00000001000901e4 fatal_error_signal + 340
8   libsystem_c.dylib             	0x00007fff84905cfa _sigtramp + 26
9   org.gnu.Emacs                 	0x00000001000281a0 handle_face_prop + 256
10  org.gnu.Emacs                 	0x0000000100028b0f handle_stop + 127
11  org.gnu.Emacs                 	0x000000010002e3b1
next_element_from_string + 273
12  org.gnu.Emacs                 	0x000000010002b2a1
get_next_display_element + 81
13  org.gnu.Emacs                 	0x000000010002c6e5
move_it_in_display_line_to + 245
14  org.gnu.Emacs                 	0x000000010002db9e move_it_to + 302
15  org.gnu.Emacs                 	0x00000001000393c1 move_it_vertically + 65
16  org.gnu.Emacs                 	0x000000010004ef95 Fwindow_end + 325
17  org.gnu.Emacs                 	0x00000001001093f1 Ffuncall + 1201
18  org.gnu.Emacs                 	0x00000001001463be Fbyte_code + 6814
19  org.gnu.Emacs                 	0x0000000100108e9c funcall_lambda + 588
20  org.gnu.Emacs                 	0x0000000100109192 Ffuncall + 594
21  org.gnu.Emacs                 	0x000000010010aade call1 + 30

-- 
Jay McCarthy <jay@cs.byu.edu>
Assistant Professor / Brigham Young University
http://faculty.cs.byu.edu/~jay

"The glory of God is Intelligence" - D&C 93

Re: Converting from OmniFocus to org-mode

[Jambunathan K]

> A section of the stack trace is at the end of this email.

Crash is really in core Emacs (re-display engine may be?) triggered by
Org related things. 

It is worth disabling font-lock in Org buffers. I am not sure whether
that would prevent you from making use of any of the Org features.

If you could bisect the Org file and narrow down the crash to a smaller
example file that would be wonderful. Can you give us an idea of number
of scheduled entries per day and how many months they typically span. I
know Org uses cache so may be the performance hit is one time only. May
be there could be a script that populates the cache as part of first
step in the migration process.

Btw, I find the use of "theoretically" a bit inconsistent in your
email. As a developer myself, I wouldn't go to the extent of a writing a
migration assistant just to reap purely "theoretical" benefits. Surely I
am nitpicking here.

Jambunathan K.

Re: Converting from OmniFocus to org-mode

[Jay McCarthy]

On Sat, Aug 6, 2011 at 4:35 AM, Jambunathan K <kjambunathan@gmail.com> wrote:
>
>> A section of the stack trace is at the end of this email.
>
> Crash is really in core Emacs (re-display engine may be?) triggered by
> Org related things.
>
> It is worth disabling font-lock in Org buffers. I am not sure whether
> that would prevent you from making use of any of the Org features.

I don't want to use it that way, but I can check if it speeds it
up.... and it doesn't make any appreciable difference on the speed and
the crash still happens.

> If you could bisect the Org file and narrow down the crash to a smaller
> example file that would be wonderful. Can you give us an idea of number
> of scheduled entries per day and how many months they typically span.

% grep -e DEADLINE -e SCHEDULED brain.org | wc -l
     462
% grep -e DEADLINE brain.org| sort | head -1
DEADLINE: <2011-08-01 17:00 +1m>
% grep -e DEADLINE brain.org| sort | tail -1
DEADLINE: <2015-09-05 00:00>
% grep -e DEADLINE brain.org| sort | uniq -c | sort | tail -10
   3 DEADLINE: <2012-04-01 17:00 +1y>
   4 DEADLINE: <2011-08-04 23:00 +1d>
   4 DEADLINE: <2011-08-05 17:00 +1d>
   4 DEADLINE: <2012-05-01 17:00 +1y>
   4 DEADLINE: <2012-09-01 17:00 +1y>
   5 DEADLINE: <2011-08-05 17:00 +1w>
   5 DEADLINE: <2011-08-06 17:00 +1w>
   7 DEADLINE: <2011-09-01 17:00>
   8 DEADLINE: <2011-08-05 21:00 +1d>
  14 DEADLINE: <2011-09-01 17:00 +1m>

Is this useful information?

> I know Org uses cache so may be the performance hit is one time only. May
> be there could be a script that populates the cache as part of first
> step in the migration process.

That sounds like a very good idea.

http://www.google.com/search?q=org-mode%20cache

Only tells me about the caching on the evaluation of code blocks, so
I'm not in a position to implement it, as someone who's never actually
used org-mode.

> Btw, I find the use of "theoretically" a bit inconsistent in your
> email. As a developer myself, I wouldn't go to the extent of a writing a
> migration assistant just to reap purely "theoretical" benefits. Surely I
> am nitpicking here.

I mean that it seems appealing, but I have no idea because I've never
used it. I need to write a migration assistant before I could even
figure out if there was a benefit.

Jay

-- 
Jay McCarthy <jay@cs.byu.edu>
Assistant Professor / Brigham Young University
http://faculty.cs.byu.edu/~jay

"The glory of God is Intelligence" - D&C 93

Re: Converting from OmniFocus to org-mode

[Bastien]

Hi Jay,

Jay McCarthy <jay.mccarthy@gmail.com> writes:

>> It is worth disabling font-lock in Org buffers. I am not sure whether
>> that would prevent you from making use of any of the Org features.
>
> I don't want to use it that way, but I can check if it speeds it
> up.... and it doesn't make any appreciable difference on the speed and
> the crash still happens.

Slowliness often comes from using many text overlays, which are used
fondamentally by outline-mode, the mode org-mode is derived from.

Turning off font-lock will not do anything about the problem you're
reporting.

Please bear with us and Emacs developers while we are improving things
is this area, this is quite a common request and should be a priority.

Best,

-- 
 Bastien

Re: Converting from OmniFocus to org-mode

[Sebastien Vauban]

Hi Jambunathan and Jay

Jambunathan K wrote:
> If you could bisect the Org file and narrow down the crash to a smaller
> example file that would be wonderful. Can you give us an idea of number of
> scheduled entries per day and how many months they typically span. I know
> Org uses cache so may be the performance hit is one time only.

As far as I know, Bastien began talking of caches with the author of
Emacs-calfw, but nothing has been done yet in that direction.

So, Jay's performance problem must be identical be it the first time or not.

> May be there could be a script that populates the cache as part of first
> step in the migration process.

Best regards,
  Seb

-- 
Sebastien Vauban

Re: Converting from OmniFocus to org-mode

[Bastien]

Hi Sébastien,

"Sebastien Vauban" <wxhgmqzgwmuf@spammotel.com> writes:

> As far as I know, Bastien began talking of caches with the author of
> Emacs-calfw, but nothing has been done yet in that direction.

Confirmed.  Once I'm done doing plumbering work and catching up emails,
I'll put a stab at this.  

Best,

-- 
 Bastien

Re: Converting from OmniFocus to org-mode

[Bastien]

Hi Jay,

Jay McCarthy <jay.mccarthy@gmail.com> writes:

> I'm an avid OmniFocus user and I've been thinking about trying
> org-mode.

Unless I misread this thread, we don't know what version of Emacs you
are using -- this is crucial information for usability problems such as
the one you are reporting.

> https://github.com/jeapostrophe/exp/blob/master/of2org.rkt

Thanks for this!  Can you write a small paragraph presenting the tool
so that I add it to http://orgmode.org/worg/org-tools/index.html?

> However, once I converted my 1.6M OmniFocus database, I got a 252K
> org-mode file. 

There should be no problem with such file -- my main org file is
currently 900k and I'm not experiencing major slowliness.  I am using
Emacs from bzr, compiled a few days ago 2011-08-13.

Best,

-- 
 Bastien

Re: Converting from OmniFocus to org-mode

[Jay McCarthy]

On Mon, Aug 15, 2011 at 2:37 AM, Bastien <bzg@altern.org> wrote:
> Hi Jay,
>
> Jay McCarthy <jay.mccarthy@gmail.com> writes:
>
>> I'm an avid OmniFocus user and I've been thinking about trying
>> org-mode.
>
> Unless I misread this thread, we don't know what version of Emacs you
> are using -- this is crucial information for usability problems such as
> the one you are reporting.

I thought that this information in the stack report would be enough:

Identifier:      org.gnu.Emacs
Version:         Version 23.3 (9.0)
OS Version:      Mac OS X 10.7 (11A511)

It's the version from http://emacsformacosx.com/

What else should I send?

>> https://github.com/jeapostrophe/exp/blob/master/of2org.rkt
>
> Thanks for this!  Can you write a small paragraph presenting the tool
> so that I add it to http://orgmode.org/worg/org-tools/index.html?

A script to convert the internal database of OmniFocus to org-mode
with minimal metadata loss. In particular, it maintains repeating
events and multi-line notes fields.

>> However, once I converted my 1.6M OmniFocus database, I got a 252K
>> org-mode file.
>
> There should be no problem with such file -- my main org file is
> currently 900k and I'm not experiencing major slowliness.  I am using
> Emacs from bzr, compiled a few days ago 2011-08-13.

Any idea what's likely to be my problem?

Jay

-- 
Jay McCarthy <jay@cs.byu.edu>
Assistant Professor / Brigham Young University
http://faculty.cs.byu.edu/~jay

"The glory of God is Intelligence" - D&C 93

Re: Converting from OmniFocus to org-mode

[suvayu ali]

On Mon, Aug 15, 2011 at 1:32 PM, Jay McCarthy <jay.mccarthy@gmail.com> wrote:
>
> What else should I send?

M-x org-version

-- 
Suvayu

Open source is the future. It sets us free.

Re: Converting from OmniFocus to org-mode

[Jay McCarthy]

It was the latest git when I sent the email (August 5th)

Jay

On Mon, Aug 15, 2011 at 6:02 AM, suvayu ali <fatkasuvayu+linux@gmail.com> wrote:
> On Mon, Aug 15, 2011 at 1:32 PM, Jay McCarthy <jay.mccarthy@gmail.com> wrote:
>>
>> What else should I send?
>
> M-x org-version
>
> --
> Suvayu
>
> Open source is the future. It sets us free.
>



-- 
Jay McCarthy <jay@cs.byu.edu>
Assistant Professor / Brigham Young University
http://faculty.cs.byu.edu/~jay

"The glory of God is Intelligence" - D&C 93

Re: Converting from OmniFocus to org-mode

[Suvayu Ali]

On Mon, 15 Aug 2011 06:09:53 -0600
Jay McCarthy <jay.mccarthy@gmail.com> wrote:

> It was the latest git when I sent the email (August 5th)
> 

Knowingly precisely which commit always helps. Hence it is customary to
post the output of the command I mentioned.

M-x org-version

or the output of 'git describe' from the shell in the org directory.

> Jay

Hope this is clearer.

-- 
Suvayu

Open source is the future. It sets us free.

Re: Converting from OmniFocus to org-mode

[Jay McCarthy]

On Mon, Aug 15, 2011 at 6:16 AM, Suvayu Ali <fatkasuvayu+linux@gmail.com> wrote:
> On Mon, 15 Aug 2011 06:09:53 -0600
> Jay McCarthy <jay.mccarthy@gmail.com> wrote:
>
>> It was the latest git when I sent the email (August 5th)
>>
>
> Knowingly precisely which commit always helps. Hence it is customary to
> post the output of the command I mentioned.
>
> M-x org-version

7.7.20.g4447

> or the output of 'git describe' from the shell in the org directory.

release_7.7-20-g44472fa


-- 
Jay McCarthy <jay@cs.byu.edu>
Assistant Professor / Brigham Young University
http://faculty.cs.byu.edu/~jay

"The glory of God is Intelligence" - D&C 93

Re: Converting from OmniFocus to org-mode

[Suvayu Ali]

On Mon, 15 Aug 2011 06:19:28 -0600
Jay McCarthy <jay.mccarthy@gmail.com> wrote:

> > or the output of 'git describe' from the shell in the org
> > directory.  
> 
> release_7.7-20-g44472fa

Does this happen with "emacs -Q -l /path/to/minimal/org-setup.el". If it
is repeatable with the above, can you try to isolate the problematic
part in your org file and post a minimal example for the devs to work
with?

There is a small FAQ entry about what is expected of a minimal example
<http://orgmode.org/worg/org-faq.html#bug-reporting>

As for a minimal org setup after 'emacs -Q', this is what I use:


(add-to-list 'load-path 
	     (expand-file-name "~/build/org-mode/lisp"))
(add-to-list 'load-path 
	     (expand-file-name "~/build/org-mode/contrib/lisp"))

;; activate org
(require 'org-install)


I hope this helps you isolating your problem and reporting back with
more details.

GL

-- 
Suvayu

Open source is the future. It sets us free.

Re: Converting from OmniFocus to org-mode

[Jay McCarthy]

This was very useful. I didn't have the problem with the minimal
setup. I then went through my global modes, turning them off until I
found that by turning off global-linum-mode it drastically improved
the performance of org-mode and removed the crash. Great! Now, I'll
try org-mode for realsies.

Jay

On Mon, Aug 15, 2011 at 6:50 AM, Suvayu Ali <fatkasuvayu+linux@gmail.com> wrote:
> On Mon, 15 Aug 2011 06:19:28 -0600
> Jay McCarthy <jay.mccarthy@gmail.com> wrote:
>
>> > or the output of 'git describe' from the shell in the org
>> > directory.
>>
>> release_7.7-20-g44472fa
>
> Does this happen with "emacs -Q -l /path/to/minimal/org-setup.el". If it
> is repeatable with the above, can you try to isolate the problematic
> part in your org file and post a minimal example for the devs to work
> with?
>
> There is a small FAQ entry about what is expected of a minimal example
> <http://orgmode.org/worg/org-faq.html#bug-reporting>
>
> As for a minimal org setup after 'emacs -Q', this is what I use:
>
>
> (add-to-list 'load-path
>             (expand-file-name "~/build/org-mode/lisp"))
> (add-to-list 'load-path
>             (expand-file-name "~/build/org-mode/contrib/lisp"))
>
> ;; activate org
> (require 'org-install)
>
>
> I hope this helps you isolating your problem and reporting back with
> more details.
>
> GL
>
> --
> Suvayu
>
> Open source is the future. It sets us free.
>



-- 
Jay McCarthy <jay@cs.byu.edu>
Assistant Professor / Brigham Young University
http://faculty.cs.byu.edu/~jay

"The glory of God is Intelligence" - D&C 93

Re: Converting from OmniFocus to org-mode

[suvayu ali]

Hi Jay,

On Mon, Aug 15, 2011 at 6:12 PM, Jay McCarthy <jay.mccarthy@gmail.com> wrote:
> This was very useful. I didn't have the problem with the minimal
> setup. I then went through my global modes, turning them off until I
> found that by turning off global-linum-mode it drastically improved
> the performance of org-mode and removed the crash. Great! Now, I'll
> try org-mode for realsies.
>

Thanks for reporting back. This has been reported at least once
before[1]. Maybe its time to document this on Worg. I hope you enjoy
orgmode (minus line numbers ;)).

To get started I would suggest going through the wonderful tutorials
on Worg[2]. Depending on what you expect from orgmode pick _one_ of
the tutorials and get started. As and when you come up with
circumstances where you need new features, you will usually find its
already available within orgmode and there is a corresponding
discussion on Worg. In the off chance the required feature is missing,
asking on the list will probably get you that feature within a few
weeks!

> Jay

GL

[1] <http://thread.gmane.org/gmane.emacs.orgmode/37860/focus=37864>
[2] <http://orgmode.org/worg/>

-- 
Suvayu

Open source is the future. It sets us free.

Re: Converting from OmniFocus to org-mode

[Nick Dokos]

Jay McCarthy <jay.mccarthy@gmail.com> wrote:

> This was very useful. I didn't have the problem with the minimal
> setup. I then went through my global modes, turning them off until I
> found that by turning off global-linum-mode it drastically improved
> the performance of org-mode and removed the crash. Great! Now, I'll
> try org-mode for realsies.


It used to be the case (and probably still is, given your experience
above) that linum-mode is expensive: if you can live without it, you'd be
much better off.

emacs tries to make it possible to go places without having to know line
numbers (e.g M-x compile will print out error messages with line numbers
but all you have to do is C-x ` to find the next error message *and* get
to the source code that caused it). That's generally a better way to go.

It's not always possible to do without line numbers (although you can
come pretty close), so there is a mode to print out the current line
number in the mode line which is not nearly as expensive:
line-number-mode. I have that on all the time and I do not notice any
performance impact.

Nick

Re: Converting from OmniFocus to org-mode

[Bastien]

Jay McCarthy <jay.mccarthy@gmail.com> writes:

> I thought that this information in the stack report would be enough:
>
> Identifier:      org.gnu.Emacs
> Version:         Version 23.3 (9.0)
> OS Version:      Mac OS X 10.7 (11A511)
>
> It's the version from http://emacsformacosx.com/

So yes, I correctly misread the thread ;)

Thanks for the additional info, glad you made some progress!

-- 
 Bastien