Re: Scaling org-mode

[d@teklibre.org (Dave Täht)]

Matt Lundin <mdl@imapmail.org> writes:

> d@teklibre.org (Dave Täht) writes:
>
>> I have really been enjoying importing my life into org-mode, which I've
>> been doing for about two months now. 
>>
>> But.
>>
>> It currently visits about 100 files and 10k of text to construct the
>> agenda. It's starting to get kind of slow and interrupt my workflow,
>> particularly the background process that scans them. 
>>
>> While the system is effectively frozen, my message buffer fills up with
>> messages about setting the flyspell dictionary to en, etc. This is quite
>> annoying with text to speech turned on. I ended up just having appts
>> spoken.
>
> My guess is that this is a flyspell problem. I only get messages about
> flyspell if I don't have aspell or the proper dictionary installed. Do
> you get these messages when you open buffers in other text modes? What

Yes. Opening up a random text file gives me:

"Local Ispell dictionary set to en" in the message buffer. It's not
annoying when opening a single file. I can hack ispell.el to suppress
the message, but I think the core problem is more starting up the aspell
process for 100+ files. Can't tell without profiling. 

When org is running in the background, I actually get two messages,
alternating, that and... Darn it, I can't get it to happen right now.

> exactly are the messages? What is your flyspell config?

It is using aspell.

I work in both spanish and english, mostly english.

;; Useful because my LANG variable is usually set to es_ES.UTF-8
;; Maybe there is another default dictionary I could set or a better way
;; to suppress the message (like, maybe suppressing the language
;; variable somewhere else), or doing this in ispell rather than flyspell

(setq flyspell-default-dictionary "en")

;; Relevant hooks
;; Maybe find something that suppresses all these hooks until the buffer
;; is actually displayed on screen would help.

(dolist (hook '(text-mode-hook))
  (add-hook hook (lambda () (flyspell-mode 1))))

(dolist (hook '(erc-mode-hook
		emacs-lisp-mode-hook
		text-mode-hook))
		(add-hook hook (lambda () (abbrev-mode 1))
		(add-hook hook (lambda () (auto-capitalization-mode 1))
))



>
>> Solution #1) cut the number of files down - is a good one. I probably
>> can cut those files easily in half right now. The problem is that I have
>> about 600 more files to import (scenes from a book), and I really like
>> the idea of being able to know what my characters are doing in 2023, and
>> separate files was kind of useful at one point.
>
> One recommendation would be to leave files that are purely notes out of
> your agenda. I noticed from your blog posts that you are adding
> timestamps to events from your novel. Are these active or inactive
> timestamps? My guess is that parsing all that data is slowing org-mode
> down. You can always create a timeline from within a particular file
> without including it in your normal agenda files. You can also create
> custom agenda views to work on different sets of org files.

Thank you for reading! I haven't got further than wanting to use
embedded logic in the book for various things like calculating relative
delta-v, I'd like to sort out some more simple stuff first (like
blogging and managing my life. I'm going to get off of blogger, that's
certain, but moving to what (and getting pretty html output) remains an
open question. 

Some percentage of 898 blog posts to convert, too. Sigh.

I'll want to keep draft blog posts in the agenda.

> http://orgmode.org/worg/org-tutorials/org-custom-agenda-commands.php

Reading... Brain dumping core... 

>
> While org-mode is exceptionally robust as a plain text tool, if you need
> to manage an ever-growing collection of thousands and thousands of
> pieces of data, you'll probably be better served by 1) using a database

Heck, no, that's what gobs of memory is for! Emacs is, for once, not the
biggest process on my system, even with over a hundred buffers loaded.

Partial top listing:

 764 d         20   0  663m 184m  13m S  0.0 19.2  10:56.43 firefox     
 3964 d         20   0  343m 164m 9112 S  0.0 17.0  11:32.16 emacs     

:)

I have tried to organize my personal collection of messes before using
everything from a zillion text files, to database based stuff like
evolution, to writing my own XML outliner, to MS-project. All fell down
somewhere.

What works best is an outliner. I really, really, really missed
MORE. Org is better than MORE in almost every way. Yep, I'm willing to
adapt to how it works and adapt it to how I work. 


> or 2) dividing your org-mode files into separate "collections" (i.e.,
> reserving your regular agenda for only a subset of org files).

That strikes me as a good option, but what I'll want to do is add the
book-mode collection of files to the normal org-mode files when in
"book-mode", so I can stay on task.

I think learning how to profile emacs is moving higher up on my list of
smart things to learn.

>
> I keep my org files lean by regularly archiving subtrees. I can always
> search the archives when I need to.

I am going to be doing that. My current issue with archiving is that I'd
like a view that shows my DONE, non-repeating tasks, so I can make sure
I've scheduled the next step for all of them before I archive them, and
also have a report of what I did for the month that I can review later.

(Maybe one exists, I'll go re-read the doc, I remember reading about it...)

I LOVE seeing the DONE tasks, it gives me a sense of accomplishment out of
a chaotic month, but yea, they need to go OOSOOM as soon as I figure out
the above.

LOTS of DONE tasks in the last two months. I feel productive.

It would be kind of cool to be able to "explode" a set of agenda items
into visible on-screen buffers.

>> so thought 2) would be to have it only attempt to construct background
>> agendas when the system is otherwise idle for a few minutes. I don't
>> know how to do that, I figure wrapping this bit with something that
>> could detect idleness instead of just running arbitrarily would be good.
>>
>>   (run-at-time nil 3600 'org-agenda-to-appt)
>>
>> don't know how to detect idleness.
>
> http://www.gnu.org/software/emacs/elisp/html_node/Idle-Timers.html

Bueno. Writing that bit of code right now. Hmmm... Maybe I can tie this into erc
which also has annoying blocking behavior on netsplits and lost connectivity.

So much depth to Emacs!

Thanks for the thoughts!

-- 
Dave Taht
http://the-edge.blogspot.com