From: John Kitchin <jkitchin@andrew.cmu.edu>
To: Karl Voit <news1142@Karl-Voit.at>
Cc: emacs-orgmode@gnu.org
Subject: Re: [Orgmode] Slow speed of week and month views
Date: Sun, 06 Aug 2017 09:47:11 -0400 [thread overview]
Message-ID: <m2efso3j1c.fsf@andrew.cmu.edu> (raw)
In-Reply-To: <2017-08-06T09-41-33@devnull.Karl-Voit.at>
Karl Voit writes:
> Hello John,
>
> Great to read your thoughts on the topic - I am a huge admirer of
> your work and we both seem to cope with similar issues with
> Org-mode.
Thanks! I am an equal admirer of the Memacs package. I think we share
some common interests there. I came across the mother of all demos
recently: http://dougengelbart.org/events/1968-demo-highlights.html. It
was inspired by the early Memex ideas.
>
> * John Kitchin <jkitchin@andrew.cmu.edu> wrote:
>>
>> One is to use the new dynamic module capability to write an org parser in
>> C, or a dedicated agenda function, which would presumably be faster than in
>> elisp. This seems hard, and for me would certainly be a multiyear project
>> I am sure! The downside of this is the need to compile the module. I don't
>> know how easy it would be to make this work across platforms with the
>> relatively easy install org-mode currently has. This could have a side
>> benefit though of a c-lib that could be used by others to expand where
>> org-mode is used.
>
> I'm not a fan of C at all but having the parser in C with the
> possibility to use this parser for external tools as well sounds
> awesome to me. After all, I've written a primitive parser for a
> sub-set of Org-mode for https://github.com/novoid/lazyblorg using
> Python.
I am also not so fond of C, but I am starting to learn it these days. I
have had several use cases where I either generate or want to consume
org in something other than Emacs. I don't know that C is the right
thing to use, maybe a Cython library would work as well with a very thin
C interface. It is on my bucket list to learn how to do something like
that.
>
>> The other way that might work is to rely more heavily on a cached version
>> of the files, perhaps in a different format than elisp, that is faster to
>> work with. The approach I have explored in this is to index org files into
>> a sqlite database. The idea then would be to generate the agenda from a sql
>> query. I use something like this already to "find stuff in orgmode
>> anywhere". One of the reasons I wrote this is the org-agenda list of files
>> isn't practical for me because my files are so scattered on my file system.
>> I had a need to be able to find TODOs in research projects in a pretty wide
>> range of locations.
>>
>> The code I use is at
>> https://github.com/jkitchin/scimax/blob/master/org-db.el, and from one
>> database I can find headlines, contacts, locations, TODO headlines across
>> my file system, all the files that contain a particular link, and my own
>> recent org files.
>
> I didn't try org-db.el yet. So far, I survived using "git grep" and
> counsel-grep [0]
The main limitation with those for me is I don't know how you would find
a headline with an EMAIL property and some tag, for example, since those
are on different lines.
>
>> I am moderately motivated to switch from sqlite to MongoDB
>
> Is org-db.el your standard way of accessing informations or do you
> use it only for occasional searches where you assume that the usual
> methods would be slow?
I think of it as my extended memory. For the buffers that are currently
open/my agenda files/recent files/project files, I usually use the
regular tools to move around and search. org-db is more for jumping to a
headline I had open last month for example, or for files that are not in
my agenda. Or to get a list of headlines with EMAIL properties and some
tag combination from any org-file I have ever opened. This is just a
query (https://github.com/jkitchin/scimax/blob/master/org-db.el#L499) on
the sqlite db, and then I can build a completion command in helm or ivy
to select them. (My elisp cached contacts code is what I usually use
https://github.com/jkitchin/scimax/blob/master/contacts.el). I don't
claim it is optimal, but it handles about 6200 contacts from 32
org-files easily!
>
>> The main point of the database was to get a query language, persistence and
>> good performance. I have also used caches to speed up using bibtex files,
>> and my org-contacts with reasonable performance. These have been all elisp,
>> with no additional dependencies. Maybe one could do something similar to
>> keep an agenda cache that is persistent and updated via hook functions.
>
> Oh yeah. My org-contacts were unusable without at least some minor
> performance improvements as well. Most important to me: improving
> manipulation of properties using [1].
The other reason I wanted org-db is also performance related. I have
close to 3400 (and growing) org files indexed in it and I want to search
across them, but I don't need them in my agenda list. I want a
structured search, e.g. to search specifically for headlines with
properties and tags, or links, or by file tags, src block, etc. I
haven't gotten completely there yet,
Another db approach is described here:
https://github.com/wvxvw/sphinx-mode. It is more comprehensive than my
sqlite implementation, and uses Sphinx on top of MySQL I think. I
haven't worked with it though.
>
> For example, org-set-property takes almost 20 seconds to give me its
> interactive input line in my main Org-mode file. This is a no-go.
> [1] helped me here a lot.
>
> [0] https://github.com/novoid/dot-emacs/blob/master/config.org#optimizing-search-methods
> [1] https://github.com/novoid/dot-emacs/blob/master/config.org#my-org-region-to-property--my-map-p
--
Professor John Kitchin
Doherty Hall A207F
Department of Chemical Engineering
Carnegie Mellon University
Pittsburgh, PA 15213
412-268-7803
@johnkitchin
http://kitchingroup.cheme.cmu.edu
next prev parent reply other threads:[~2017-08-06 13:47 UTC|newest]
Thread overview: 19+ messages / expand[flat|nested] mbox.gz Atom feed top
2017-08-04 15:14 [Orgmode] Slow speed of week and month views Christoph Groth
2017-08-05 3:07 ` Adam Porter
2017-08-05 21:09 ` Karl Voit
2017-08-05 22:17 ` John Kitchin
2017-08-05 22:33 ` Nicolas Goaziou
2017-08-06 2:28 ` [Orgmode] " Tim Cross
2017-08-06 2:36 ` Adam Porter
2017-08-06 7:54 ` Karl Voit
2017-08-06 13:47 ` John Kitchin [this message]
2017-08-05 22:35 ` Nicolas Goaziou
2017-08-06 8:00 ` Karl Voit
2017-08-06 8:15 ` Nicolas Goaziou
2017-08-06 9:01 ` Karl Voit
2017-08-06 9:13 ` Adam Porter
2017-08-06 10:03 ` Karl Voit
2017-08-06 10:28 ` Adam Porter
2017-08-06 9:26 ` Nicolas Goaziou
2017-08-06 10:18 ` Karl Voit
2017-08-07 11:41 ` Nicolas Goaziou
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=m2efso3j1c.fsf@andrew.cmu.edu \
--to=jkitchin@andrew.cmu.edu \
--cc=emacs-orgmode@gnu.org \
--cc=news1142@Karl-Voit.at \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
Code repositories for project(s) associated with this external index
https://git.savannah.gnu.org/cgit/emacs.git
https://git.savannah.gnu.org/cgit/emacs/org-mode.git
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.