* Scaling org-mode @ 2009-09-13 5:45 Dave Täht 2009-09-13 14:04 ` Matt Lundin ` (2 more replies) 0 siblings, 3 replies; 7+ messages in thread From: Dave Täht @ 2009-09-13 5:45 UTC (permalink / raw To: emacs-orgmode -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 I have really been enjoying importing my life into org-mode, which I've been doing for about two months now. But. It currently visits about 100 files and 10k of text to construct the agenda. It's starting to get kind of slow and interrupt my workflow, particularly the background process that scans them. While the system is effectively frozen, my message buffer fills up with messages about setting the flyspell dictionary to en, etc. This is quite annoying with text to speech turned on. I ended up just having appts spoken. ;; quick hack for saying announcements, need more thought turned into it. (when window-system (defun dtaht/say-stuff (id msg &optional delay vattrib hattrib font) "Speak a message msg. Currently requires say.el" (unless vattrib (setq vattrib "top")) (unless hattrib (setq hattrib "right")) (unless delay (setq delay 5000)) (unless font (setq font "Arial 12")) (save-window-excursion (say (dtaht/ssml-escape msg) )))) ;; my personal fav, run every 15 minutes (defun nag-timer () "Nag me when there isn't a clock running" (interactive) (unless (marker-buffer org-clock-marker) (say "Are you mating now?"))) It gets a bit chunky even when all the org files are in memory (and I have gobs of memory). Solution #1) cut the number of files down - is a good one. I probably can cut those files easily in half right now. The problem is that I have about 600 more files to import (scenes from a book), and I really like the idea of being able to know what my characters are doing in 2023, and separate files was kind of useful at one point. That's a couple hundred k of text (what I have in there now is mostly "normal" items for managing my personal and professional life) Solution #2) Make org-mode faster. I am compiling and installing org-mode from git at the moment. So it's compiled. I have a lot of hooks installed for text buffers - At minimum, auto-capitalize, flyspell, yas, wrap, and abbrev. Most (all?) of those are compiled, too, but aren't needed to be run or initialized when merely being visited. so thought 1) would be to come up with some sort of text mode hook that only hooks in when a human is viewing or editing the buffer, not when it is happening programmatically. so thought 2) would be to have it only attempt to construct background agendas when the system is otherwise idle for a few minutes. I don't know how to do that, I figure wrapping this bit with something that could detect idleness instead of just running arbitrarily would be good. (run-at-time nil 3600 'org-agenda-to-appt) don't know how to detect idleness. And thought 0) would be to understand emacs well enough to profile what's really going on as maybe "starting a new flyspell process" is not really the problem but a symptom, but I don't (yet). (suggestions desired. I have read up a bit on emacs profiling) I am deeply grateful for the existence of org-mode and the fine work going into it. I wrote a nice blog entry about it recently. http://the-edge.blogspot.com/2009/08/going-retro-re-adopting-emacs.html TIA. - -- Dave Taht http://the-edge.blogspot.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (GNU/Linux) Comment: Processed by Mailcrypt 3.5.8+ <http://mailcrypt.sourceforge.net/> iEYEARECAAYFAkqshuUACgkQpdejJcOV4uRkTwCgyz5IMWJTzCHa8CmUAgU/fyIa cvwAoJKQ5Gr+1vlSAbEfbKob76xJIvB8 =/Xdv -----END PGP SIGNATURE----- ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: Scaling org-mode 2009-09-13 5:45 Scaling org-mode Dave Täht @ 2009-09-13 14:04 ` Matt Lundin 2009-09-13 17:25 ` Dave Täht 2009-09-13 16:52 ` Nick Dokos 2009-09-13 23:30 ` Productiviy tools (was: Scaling org-mode) Daniel Clemente 2 siblings, 1 reply; 7+ messages in thread From: Matt Lundin @ 2009-09-13 14:04 UTC (permalink / raw To: Dave Täht; +Cc: emacs-orgmode d@teklibre.org (Dave Täht) writes: > I have really been enjoying importing my life into org-mode, which I've > been doing for about two months now. > > But. > > It currently visits about 100 files and 10k of text to construct the > agenda. It's starting to get kind of slow and interrupt my workflow, > particularly the background process that scans them. > > While the system is effectively frozen, my message buffer fills up with > messages about setting the flyspell dictionary to en, etc. This is quite > annoying with text to speech turned on. I ended up just having appts > spoken. My guess is that this is a flyspell problem. I only get messages about flyspell if I don't have aspell or the proper dictionary installed. Do you get these messages when you open buffers in other text modes? What exactly are the messages? What is your flyspell config? > Solution #1) cut the number of files down - is a good one. I probably > can cut those files easily in half right now. The problem is that I have > about 600 more files to import (scenes from a book), and I really like > the idea of being able to know what my characters are doing in 2023, and > separate files was kind of useful at one point. One recommendation would be to leave files that are purely notes out of your agenda. I noticed from your blog posts that you are adding timestamps to events from your novel. Are these active or inactive timestamps? My guess is that parsing all that data is slowing org-mode down. You can always create a timeline from within a particular file without including it in your normal agenda files. You can also create custom agenda views to work on different sets of org files. http://orgmode.org/worg/org-tutorials/org-custom-agenda-commands.php While org-mode is exceptionally robust as a plain text tool, if you need to manage an ever-growing collection of thousands and thousands of pieces of data, you'll probably be better served by 1) using a database or 2) dividing your org-mode files into separate "collections" (i.e., reserving your regular agenda for only a subset of org files). I keep my org files lean by regularly archiving subtrees. I can always search the archives when I need to. > so thought 2) would be to have it only attempt to construct background > agendas when the system is otherwise idle for a few minutes. I don't > know how to do that, I figure wrapping this bit with something that > could detect idleness instead of just running arbitrarily would be good. > > (run-at-time nil 3600 'org-agenda-to-appt) > > don't know how to detect idleness. http://www.gnu.org/software/emacs/elisp/html_node/Idle-Timers.html Best, Matt ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: Scaling org-mode 2009-09-13 14:04 ` Matt Lundin @ 2009-09-13 17:25 ` Dave Täht 2009-09-13 18:58 ` Dave Täht 0 siblings, 1 reply; 7+ messages in thread From: Dave Täht @ 2009-09-13 17:25 UTC (permalink / raw To: Matt Lundin; +Cc: emacs-orgmode Matt Lundin <mdl@imapmail.org> writes: > d@teklibre.org (Dave Täht) writes: > >> I have really been enjoying importing my life into org-mode, which I've >> been doing for about two months now. >> >> But. >> >> It currently visits about 100 files and 10k of text to construct the >> agenda. It's starting to get kind of slow and interrupt my workflow, >> particularly the background process that scans them. >> >> While the system is effectively frozen, my message buffer fills up with >> messages about setting the flyspell dictionary to en, etc. This is quite >> annoying with text to speech turned on. I ended up just having appts >> spoken. > > My guess is that this is a flyspell problem. I only get messages about > flyspell if I don't have aspell or the proper dictionary installed. Do > you get these messages when you open buffers in other text modes? What Yes. Opening up a random text file gives me: "Local Ispell dictionary set to en" in the message buffer. It's not annoying when opening a single file. I can hack ispell.el to suppress the message, but I think the core problem is more starting up the aspell process for 100+ files. Can't tell without profiling. When org is running in the background, I actually get two messages, alternating, that and... Darn it, I can't get it to happen right now. > exactly are the messages? What is your flyspell config? It is using aspell. I work in both spanish and english, mostly english. ;; Useful because my LANG variable is usually set to es_ES.UTF-8 ;; Maybe there is another default dictionary I could set or a better way ;; to suppress the message (like, maybe suppressing the language ;; variable somewhere else), or doing this in ispell rather than flyspell (setq flyspell-default-dictionary "en") ;; Relevant hooks ;; Maybe find something that suppresses all these hooks until the buffer ;; is actually displayed on screen would help. (dolist (hook '(text-mode-hook)) (add-hook hook (lambda () (flyspell-mode 1)))) (dolist (hook '(erc-mode-hook emacs-lisp-mode-hook text-mode-hook)) (add-hook hook (lambda () (abbrev-mode 1)) (add-hook hook (lambda () (auto-capitalization-mode 1)) )) > >> Solution #1) cut the number of files down - is a good one. I probably >> can cut those files easily in half right now. The problem is that I have >> about 600 more files to import (scenes from a book), and I really like >> the idea of being able to know what my characters are doing in 2023, and >> separate files was kind of useful at one point. > > One recommendation would be to leave files that are purely notes out of > your agenda. I noticed from your blog posts that you are adding > timestamps to events from your novel. Are these active or inactive > timestamps? My guess is that parsing all that data is slowing org-mode > down. You can always create a timeline from within a particular file > without including it in your normal agenda files. You can also create > custom agenda views to work on different sets of org files. Thank you for reading! I haven't got further than wanting to use embedded logic in the book for various things like calculating relative delta-v, I'd like to sort out some more simple stuff first (like blogging and managing my life. I'm going to get off of blogger, that's certain, but moving to what (and getting pretty html output) remains an open question. Some percentage of 898 blog posts to convert, too. Sigh. I'll want to keep draft blog posts in the agenda. > http://orgmode.org/worg/org-tutorials/org-custom-agenda-commands.php Reading... Brain dumping core... > > While org-mode is exceptionally robust as a plain text tool, if you need > to manage an ever-growing collection of thousands and thousands of > pieces of data, you'll probably be better served by 1) using a database Heck, no, that's what gobs of memory is for! Emacs is, for once, not the biggest process on my system, even with over a hundred buffers loaded. Partial top listing: 764 d 20 0 663m 184m 13m S 0.0 19.2 10:56.43 firefox 3964 d 20 0 343m 164m 9112 S 0.0 17.0 11:32.16 emacs :) I have tried to organize my personal collection of messes before using everything from a zillion text files, to database based stuff like evolution, to writing my own XML outliner, to MS-project. All fell down somewhere. What works best is an outliner. I really, really, really missed MORE. Org is better than MORE in almost every way. Yep, I'm willing to adapt to how it works and adapt it to how I work. > or 2) dividing your org-mode files into separate "collections" (i.e., > reserving your regular agenda for only a subset of org files). That strikes me as a good option, but what I'll want to do is add the book-mode collection of files to the normal org-mode files when in "book-mode", so I can stay on task. I think learning how to profile emacs is moving higher up on my list of smart things to learn. > > I keep my org files lean by regularly archiving subtrees. I can always > search the archives when I need to. I am going to be doing that. My current issue with archiving is that I'd like a view that shows my DONE, non-repeating tasks, so I can make sure I've scheduled the next step for all of them before I archive them, and also have a report of what I did for the month that I can review later. (Maybe one exists, I'll go re-read the doc, I remember reading about it...) I LOVE seeing the DONE tasks, it gives me a sense of accomplishment out of a chaotic month, but yea, they need to go OOSOOM as soon as I figure out the above. LOTS of DONE tasks in the last two months. I feel productive. It would be kind of cool to be able to "explode" a set of agenda items into visible on-screen buffers. >> so thought 2) would be to have it only attempt to construct background >> agendas when the system is otherwise idle for a few minutes. I don't >> know how to do that, I figure wrapping this bit with something that >> could detect idleness instead of just running arbitrarily would be good. >> >> (run-at-time nil 3600 'org-agenda-to-appt) >> >> don't know how to detect idleness. > > http://www.gnu.org/software/emacs/elisp/html_node/Idle-Timers.html Bueno. Writing that bit of code right now. Hmmm... Maybe I can tie this into erc which also has annoying blocking behavior on netsplits and lost connectivity. So much depth to Emacs! Thanks for the thoughts! -- Dave Taht http://the-edge.blogspot.com ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: Re: Scaling org-mode 2009-09-13 17:25 ` Dave Täht @ 2009-09-13 18:58 ` Dave Täht 0 siblings, 0 replies; 7+ messages in thread From: Dave Täht @ 2009-09-13 18:58 UTC (permalink / raw To: Matt Lundin; +Cc: emacs-orgmode d@teklibre.org (Dave Täht) writes: > Matt Lundin <mdl@imapmail.org> writes: > >> d@teklibre.org (Dave Täht) writes: > >>> so thought 2) would be to have it only attempt to construct background >>> agendas when the system is otherwise idle for a few minutes. I don't >>> know how to do that, I figure wrapping this bit with something that >>> could detect idleness instead of just running arbitrarily would be good. >>> >>> (run-at-time nil 3600 'org-agenda-to-appt) >>> >>> don't know how to detect idleness. >> >> http://www.gnu.org/software/emacs/elisp/html_node/Idle-Timers.html > > Bueno. Writing that bit of code right now. Hmmm... Maybe I can tie this into erc > which also has annoying blocking behavior on netsplits and lost connectivity. First cut at making org more idle friendly. It appears to be working... but I've been typing too much to really notice. :) See: http://pastebin.com/dbb75ca for details. I have not written much emacs lisp for years, so review would be nice. I changed it to actually try to reconstruct the agenda every 10 minutes in this kinder, gentler fashion. Still have a bit more to do but this takes out the main problem. -- Dave Taht http://the-edge.blogspot.com ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: Scaling org-mode 2009-09-13 5:45 Scaling org-mode Dave Täht 2009-09-13 14:04 ` Matt Lundin @ 2009-09-13 16:52 ` Nick Dokos 2009-09-13 23:30 ` Productiviy tools (was: Scaling org-mode) Daniel Clemente 2 siblings, 0 replies; 7+ messages in thread From: Nick Dokos @ 2009-09-13 16:52 UTC (permalink / raw To: Dave =?utf-8?Q?T=C3=A4ht?=; +Cc: emacs-orgmode Dave Täht <d@teklibre.org> wrote: > ... > so thought 2) would be to have it only attempt to construct background > agendas when the system is otherwise idle for a few minutes. I don't > know how to do that, I figure wrapping this bit with something that > could detect idleness instead of just running arbitrarily would be good. > > (run-at-time nil 3600 'org-agenda-to-appt) > > don't know how to detect idleness. > [Apologies if this is off-topic - I haven't read your message carefully, as I was catching up on email between jobs.] Not sure whether it'll work any better, but there is run-with-idle-timer. C-h f run-with-idle-timer <RET> says: ,---- | run-with-idle-timer is an interactive compiled Lisp function in | `timer.el'. | | (run-with-idle-timer SECS REPEAT FUNCTION &rest ARGS) | | Perform an action the next time Emacs is idle for SECS seconds. | The action is to call FUNCTION with arguments ARGS. | SECS may be an integer, a floating point number, or the internal | time format (HIGH LOW USECS) returned by, e.g., `current-idle-time'. | If Emacs is currently idle, and has been idle for N seconds (N < SECS), | then it will call FUNCTION in SECS - N seconds from now. | | If REPEAT is non-nil, do the action each time Emacs has been idle for | exactly SECS seconds (that is, only once for each time Emacs becomes idle). | | This function returns a timer object which you can use in `cancel-timer'. `---- HTH, Nick ^ permalink raw reply [flat|nested] 7+ messages in thread
* Productiviy tools (was: Scaling org-mode) 2009-09-13 5:45 Scaling org-mode Dave Täht 2009-09-13 14:04 ` Matt Lundin 2009-09-13 16:52 ` Nick Dokos @ 2009-09-13 23:30 ` Daniel Clemente 2009-09-14 0:21 ` Productiviy tools Dave Täht 2 siblings, 1 reply; 7+ messages in thread From: Daniel Clemente @ 2009-09-13 23:30 UTC (permalink / raw To: Dave Täht; +Cc: emacs-orgmode Off-topic. El dom, sep 13 2009 a les 07:45, Dave Täht va escriure: > ;; my personal fav, run every 15 minutes > > (defun nag-timer () "Nag me when there isn't a clock running" > (interactive) > (unless (marker-buffer org-clock-marker) > (say "Are you mating now?"))) > I like this very much and have started using it; let's see how annoying it can be. Do you really clock all the time you have Emacs open? That will give very complete statistics about daily computer usage… if only you don't end up clocking everything into a general task „* do some things“. I have since long thought of more utilities like this, which watch my work habits and help me correct them in the ways I defined beforehand. It would be something like my org-boss and include: - warn when I'm not clocking anything (possibly do this only on work hours, not at home) - check that each work day I work the hours I should, no less - warn when some tasks or deadlines start to seem difficult to complete on time: - e.g. if there are still 30 predicted hours but the deadline is tomorrow (so you won't be able to do those 30 hours) - or if I am being too slow (e.g. if after 1h working at a 4h task I am still at 10%. To be on schedule I should have been at 25%) - motivate me positively when I complete tasks faster than planned - help me find the effort estimates which proved wrong (because I spent more time than planned) - warn when I have too many scheduled tasks for today in my agenda (I should reschedule them) - complain if I have many same-level tasks and I haven't assigned priorities to them - complain if I hadn't estimated the effort of task which has taken a lot of time - … I see there is much work to do. Many productivity improvements are personal, so a single mode can't match all corrective needs. A single file with a collection of working functions would be better; then users can adapt to their needs the functions they want. How does this utopia sound? I alone can't develop this in time, but: if we put a file in Worg or contrib/, could we collect all our productiviy improvement tools and ideas? -- Daniel ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: Productiviy tools 2009-09-13 23:30 ` Productiviy tools (was: Scaling org-mode) Daniel Clemente @ 2009-09-14 0:21 ` Dave Täht 0 siblings, 0 replies; 7+ messages in thread From: Dave Täht @ 2009-09-14 0:21 UTC (permalink / raw To: Daniel Clemente; +Cc: emacs-orgmode Daniel Clemente <n142857@gmail.com> writes: > Off-topic. > > El dom, sep 13 2009 a les 07:45, Dave Täht va escriure: >> ;; my personal fav, run every 15 minutes >> >> (defun nag-timer () "Nag me when there isn't a clock running" >> (interactive) >> (unless (marker-buffer org-clock-marker) >> (say "Are you mating now?"))) >> > > I like this very much and have started using it; let's see how > annoying it can be. Well, say is a wrapper around a call to the cepstral speech synth, it used to talk to espeak. Both are having grave difficulties sending stuff over ESD to my nokia 770 with the latest pulseaudio in ubuntu 9.04. I really like using ESD. I can listen to good quality music on that handheld, AND still have various announcements play over the music. I can wander the house on headphones and still function. I'm about ready to dump pulseaudio and write a wrapper for cepstral to send stuff directly to esd with no intermediaries, and/or tie cepstral into espeak properly. I am not big on visual, pop-up reminders, or things that beep or bong. You really can (with procmail + ssml) convert the old "You have mail" Aol thing into "You have mail from John Doe", if you want. > Do you really clock all the time you have Emacs open? That will give > very complete statistics about daily computer usage… if only you > don't end up clocking everything into a general task „* do some > things“. Not yet, but I'm getting better at it. I had to move more and more of my life into emacs to be able to do it, and the big piece missing is tracking web usage. I have really begun to hate the default black on white display of most web pages, and am considering adopting conqueror I'd like to have that general task "do some things" be a bit bucket for a daily: "You spent X time editing Y file" "You spent X time in an erc window" "You spent X time in email" (actually, finer grained would be good) "You spent X time not on the clock" Which I could then re-allocate over my current projects, but am very unsure as to how to go about it. I've seen a few tools that track Xwindow usage.... > I have since long thought of more utilities like this, which watch > my work habits and help me correct them in the ways I defined > beforehand. It would be something like my org-boss and include: I like to think of the voices I use as virtual secretaries, not bosses. They work for me, not vice versa. If I ever abstract the functionalities of the various say scripts enough, I'll end up calling the package "majel" after Majel Barret and come up with some suitable backronym. Majel would gather up stuff overnight and tell me in the morning that VIPs Bob, Doc, and Joe sent me mail, that the surf is up, and winds offshore, and yesterday I didn't do a drop of billable work, and tell me a joke, and tomorrow I have to pay some bills, while I blurrily consumed coffee... > - warn when I'm not clocking anything (possibly do this only on work > hours, not at home) Yes, my previous mail here has the start at an attempt to divide up work and play more than I do currently. Don't want to be nagged 24 hours a day. > - check that each work day I work the hours I should, no less Heh, for me, I also need reminders to NOT work MORE hours than I should, and make time for the fun stuff, like playing music and surfing. I really have to get the tide schedule into this thing somehow.... >- warn when some tasks or deadlines start to seem > difficult to complete on time: - e.g. if there are still 30 predicted > hours but the deadline is tomorrow (so you won't be able to do those > 30 hours) - or if I am being too slow (e.g. if after 1h working at a > 4h task I am still at 10%. To be on schedule I should have been at > 25%) Related to that I have a backlog of tasks that I would like to bulk reschedule so I stop seeing them. I have about 25, now, and I'm moving next week, so I'm just not going to get them done and have to push them all into the future somehow. Also related to that would be some sort of ms-project-like load leveler, where I could see that I had 100 hrs of stuff scheduled for next week and have an easier way of sorting it out. Similarly, a bulk estimator to mark a set of tasks and put in estimates for them all, fast. (I'm pretty sure I can do this last already. Again, I'm only 2 months into org-mode, many things, like publishing, remain a mystery. I had an abortive experience with blorg this afternoon, for example. I liked what I saw of blorg, didn't realize it had been discontinued, don't want to run ruby for a blog server, and haven't figured out if org-publish does enough of what I want) > - motivate me positively when I complete tasks faster than > planned - help me find the effort estimates which proved wrong > (because I spent more time than planned) - warn when I have too many > scheduled tasks for today in my agenda (I should reschedule them) - > complain if I have many same-level tasks and I haven't assigned > priorities to them - complain if I hadn't estimated the effort of task > which has taken a lot of time - … All good things. A few more priority levels than 3 would be good. Probably. > > I see there is much work to do. Many productivity improvements are > personal, so a single mode can't match all corrective needs. A > single file with a collection of working functions would be better; > then users can adapt to their needs the functions they want. > > > How does this utopia sound? I alone can't develop this in time, > but: if we put a file in Worg or contrib/, could we collect all our > productiviy improvement tools and ideas? Sure. At the moment though, the only big thing I was thinking of doing was full blown SSML (speech synthesis markup language) support for org, based on the existing html mode, using emphasis to handle various levels of ***** and using paragraph breaks, and eliding [[][]] stuff more correctly than what I have now. > > > -- Daniel > -- Dave Taht http://the-edge.blogspot.com ^ permalink raw reply [flat|nested] 7+ messages in thread
end of thread, other threads:[~2009-09-14 0:22 UTC | newest] Thread overview: 7+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2009-09-13 5:45 Scaling org-mode Dave Täht 2009-09-13 14:04 ` Matt Lundin 2009-09-13 17:25 ` Dave Täht 2009-09-13 18:58 ` Dave Täht 2009-09-13 16:52 ` Nick Dokos 2009-09-13 23:30 ` Productiviy tools (was: Scaling org-mode) Daniel Clemente 2009-09-14 0:21 ` Productiviy tools Dave Täht
Code repositories for project(s) associated with this external index https://git.savannah.gnu.org/cgit/emacs.git https://git.savannah.gnu.org/cgit/emacs/org-mode.git This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.