Improving org-contacts performance (and state of development in general)

Improving org-contacts performance (and state of development in general)

[Daryl Manning]

[-- Attachment #1: Type: text/plain, Size: 1015 bytes --]

Strangely, I've come to rely over the last year on org-contacts as a
lightweight, taggable CRM. However, as the file and C-z notes have grown,
performance has really started to drag. I know people have used various
schemes (caching) etc to try to improve performance and the like, but
updates to the file are taking a solid 5 seconds now when making major
updates and moving tags around.

Is there a solid, forked branch anywhere that focuses on enhancing
performance anywhere? I'm tempted to wade in and add features and
improvements myself but my elisp-fu is dodgy at best (more golang these
days.).

I'd be interested in what people are doing to speed it up (and if it is
under anything like active development for improvements. It does feel super
handy, and feels like it just needs a performance and more modern features
overhaul - more on interoperability and less on in-emacs interoperability.).

Would love to hear what people have done overall workflow wise if they are
using it seriously.

thanks,
Daryl.

[-- Attachment #2: Type: text/html, Size: 1187 bytes --]

Re: Improving org-contacts performance (and state of development in general)

[Bastien]

Hi Daryl,

Daryl Manning <dwm+orgmode@wakatara.com> writes:

> Is there a solid, forked branch anywhere that focuses on enhancing
> performance anywhere?

Note that I'm aware of, but I don't use org-contacts.el, so maybe
other will have more information.

> I'm tempted to wade in and add features and improvements myself but
> my elisp-fu is dodgy at best (more golang these days.).

I'm copying Julien, the original author of org-contacts.el, to check
if he plans to maintain org-contacts.el further and if he's okay for
someone to take over.

That someone could be you: better to have a maintainer than no one.

Also, org-contacts.el is part of the "contrib/" directory in Org's
repository and soon after 9.4 we will strip this directory out of the
org-mode.git repo, so having someone taking care of individual contrib
packages would be really appreciated.

> I'd be interested in what people are doing to speed it up (and if it
> is under anything like active development for improvements. It does
> feel super handy, and feels like it just needs a performance and more
> modern features overhaul - more on interoperability and less on
> in-emacs interoperability.).

I'm afraid most tricks will be general Org tricks, no directly related
to the code in org-contacts.el -- but perhaps I'm wrong.

Best,

-- 
 Bastien

Re: Improving org-contacts performance (and state of development in general)

[Ihor Radchenko]

> However, as the file and C-z notes have grown,
> performance has really started to drag. I know people have used various
> schemes (caching) etc to try to improve performance and the like, but
> updates to the file are taking a solid 5 seconds now when making major
> updates and moving tags around.

Could you provide some examples what exactly is being slow?
Maybe my WIP work on improving performance on large org files [1] might
help. 

Best,
Ihor

[1] https://www.mail-archive.com/emacs-orgmode@gnu.org/msg127740.html


Daryl Manning <dwm+orgmode@wakatara.com> writes:

> Strangely, I've come to rely over the last year on org-contacts as a
> lightweight, taggable CRM. However, as the file and C-z notes have grown,
> performance has really started to drag. I know people have used various
> schemes (caching) etc to try to improve performance and the like, but
> updates to the file are taking a solid 5 seconds now when making major
> updates and moving tags around.
>
> Is there a solid, forked branch anywhere that focuses on enhancing
> performance anywhere? I'm tempted to wade in and add features and
> improvements myself but my elisp-fu is dodgy at best (more golang these
> days.).
>
> I'd be interested in what people are doing to speed it up (and if it is
> under anything like active development for improvements. It does feel super
> handy, and feels like it just needs a performance and more modern features
> overhaul - more on interoperability and less on in-emacs interoperability.).
>
> Would love to hear what people have done overall workflow wise if they are
> using it seriously.
>
> thanks,
> Daryl.

Re: Improving org-contacts performance (and state of development in general)

[Daryl Manning]

[-- Attachment #1: Type: text/plain, Size: 2707 bytes --]

Primary examples would be adding a note (CTRL-z) or changing a tag on a
person and then having org-agenda update that. I am assuming it is because
the entire file needs to be parsed rather than say, some index of entries.

(so perhaps I mischaracterized org-contacts as being slow versus its
interaction with other programs.)

(for search I use swiper which is very efficient for searching the file
whenI need it.).

tho quite interested in seeing what perf enhancements you've done on large
org files would be interesting.

Daryl.
PS> As an outside feature though, interoperability of the org-contact
formats with other operating system address books, most notable gnome
contacts/evolution, goog contacts, and OSX address book would be high on my
list in terms of improving org-contacts though. (eg, raw, structued info in
all address books, and say perhaps notes or similar maintained and synced
in ome manner.



On Mon, Sep 7, 2020 at 10:27 AM Ihor Radchenko <yantar92@gmail.com> wrote:

> > However, as the file and C-z notes have grown,
> > performance has really started to drag. I know people have used various
> > schemes (caching) etc to try to improve performance and the like, but
> > updates to the file are taking a solid 5 seconds now when making major
> > updates and moving tags around.
>
> Could you provide some examples what exactly is being slow?
> Maybe my WIP work on improving performance on large org files [1] might
> help.
>
> Best,
> Ihor
>
> [1] https://www.mail-archive.com/emacs-orgmode@gnu.org/msg127740.html
>
>
> Daryl Manning <dwm+orgmode@wakatara.com> writes:
>
> > Strangely, I've come to rely over the last year on org-contacts as a
> > lightweight, taggable CRM. However, as the file and C-z notes have grown,
> > performance has really started to drag. I know people have used various
> > schemes (caching) etc to try to improve performance and the like, but
> > updates to the file are taking a solid 5 seconds now when making major
> > updates and moving tags around.
> >
> > Is there a solid, forked branch anywhere that focuses on enhancing
> > performance anywhere? I'm tempted to wade in and add features and
> > improvements myself but my elisp-fu is dodgy at best (more golang these
> > days.).
> >
> > I'd be interested in what people are doing to speed it up (and if it is
> > under anything like active development for improvements. It does feel
> super
> > handy, and feels like it just needs a performance and more modern
> features
> > overhaul - more on interoperability and less on in-emacs
> interoperability.).
> >
> > Would love to hear what people have done overall workflow wise if they
> are
> > using it seriously.
> >
> > thanks,
> > Daryl.
>

[-- Attachment #2: Type: text/html, Size: 3544 bytes --]

Re: Improving org-contacts performance (and state of development in general)

[Julien Danjou]

On Sun, Sep 06 2020, Bastien wrote:

Hi everyone,

> I'm copying Julien, the original author of org-contacts.el, to check
> if he plans to maintain org-contacts.el further and if he's okay for
> someone to take over.

I don't plan to do anything, feel free to take over!

Cheers,
-- 
Julien Danjou
# Free Software hacker
# https://julien.danjou.info

Re: Improving org-contacts performance (and state of development in general)

[Ihor Radchenko]

> tho quite interested in seeing what perf enhancements you've done on large
> org files would be interesting.

That's primarily a one single enhancement - use text properties instead
of overlays to hide/fold text. Overlays are slow - every time Emacs need
to move point across hidden overlays, it takes O(N_overlays). The
problem especially affects buffers with many property drawers (every
drawer may require an individual overlay) - should be the case for
org-contacts files. 

> Primary examples would be adding a note (CTRL-z) or changing a tag on a
> person and then having org-agenda update that. I am assuming it is because
> the entire file needs to be parsed rather than say, some index of entries.

I did some benchmarks on my agendas with and without my enhancements -
my main agenda builds in around 7sec now in comparison with 18-20sec
when using overlays. Though 20sec benchmark was not on current master,
there were several commits reducing the number of overlays in some cases
after I did the benchmark.

Also, you may consider native-comp branch. It can push the speed even
further. My agenda only took 3-4 seconds on native-comp Emacs branch.

> PS> As an outside feature though, interoperability of the org-contact
> formats with other operating system address books, most notable gnome
> contacts/evolution, goog contacts, and OSX address book would be high on my
> list in terms of improving org-contacts though. (eg, raw, structued info in
> all address books, and say perhaps notes or similar maintained and synced
> in ome manner.

I guess it is a job for [future] ox-*.el packages.

Best,
Ihor

Daryl Manning <dwm+orgmode@wakatara.com> writes:

> Primary examples would be adding a note (CTRL-z) or changing a tag on a
> person and then having org-agenda update that. I am assuming it is because
> the entire file needs to be parsed rather than say, some index of entries.
>
> (so perhaps I mischaracterized org-contacts as being slow versus its
> interaction with other programs.)
>
> (for search I use swiper which is very efficient for searching the file
> whenI need it.).
>
> tho quite interested in seeing what perf enhancements you've done on large
> org files would be interesting.
>
> Daryl.
> PS> As an outside feature though, interoperability of the org-contact
> formats with other operating system address books, most notable gnome
> contacts/evolution, goog contacts, and OSX address book would be high on my
> list in terms of improving org-contacts though. (eg, raw, structued info in
> all address books, and say perhaps notes or similar maintained and synced
> in ome manner.
>
>
>
> On Mon, Sep 7, 2020 at 10:27 AM Ihor Radchenko <yantar92@gmail.com> wrote:
>
>> > However, as the file and C-z notes have grown,
>> > performance has really started to drag. I know people have used various
>> > schemes (caching) etc to try to improve performance and the like, but
>> > updates to the file are taking a solid 5 seconds now when making major
>> > updates and moving tags around.
>>
>> Could you provide some examples what exactly is being slow?
>> Maybe my WIP work on improving performance on large org files [1] might
>> help.
>>
>> Best,
>> Ihor
>>
>> [1] https://www.mail-archive.com/emacs-orgmode@gnu.org/msg127740.html
>>
>>
>> Daryl Manning <dwm+orgmode@wakatara.com> writes:
>>
>> > Strangely, I've come to rely over the last year on org-contacts as a
>> > lightweight, taggable CRM. However, as the file and C-z notes have grown,
>> > performance has really started to drag. I know people have used various
>> > schemes (caching) etc to try to improve performance and the like, but
>> > updates to the file are taking a solid 5 seconds now when making major
>> > updates and moving tags around.
>> >
>> > Is there a solid, forked branch anywhere that focuses on enhancing
>> > performance anywhere? I'm tempted to wade in and add features and
>> > improvements myself but my elisp-fu is dodgy at best (more golang these
>> > days.).
>> >
>> > I'd be interested in what people are doing to speed it up (and if it is
>> > under anything like active development for improvements. It does feel
>> super
>> > handy, and feels like it just needs a performance and more modern
>> features
>> > overhaul - more on interoperability and less on in-emacs
>> interoperability.).
>> >
>> > Would love to hear what people have done overall workflow wise if they
>> are
>> > using it seriously.
>> >
>> > thanks,
>> > Daryl.
>>

Re: Improving org-contacts performance (and state of development in general)

[Bastien]

Julien Danjou <julien@danjou.info> writes:

>> I'm copying Julien, the original author of org-contacts.el, to check
>> if he plans to maintain org-contacts.el further and if he's okay for
>> someone to take over.
>
> I don't plan to do anything, feel free to take over!

Thanks for confirming - Daryl, don't hesitate!

-- 
 Bastien

Re: Improving org-contacts performance (and state of development in general)

[Bastien]

Ihor Radchenko <yantar92@gmail.com> writes:

> That's primarily a one single enhancement - use text properties
> instead of overlays to hide/fold text.

... or improve the speed for overlays in Emacs :)

-- 
 Bastien

Re: Improving org-contacts performance (and state of development in general)

[Ihor Radchenko]

> ... or improve the speed for overlays in Emacs :)

My Emacs-C-fu is not good enough ¯ \ _ (ツ) _ / ¯

Bastien <bzg@gnu.org> writes:

> Ihor Radchenko <yantar92@gmail.com> writes:
>
>> That's primarily a one single enhancement - use text properties
>> instead of overlays to hide/fold text.
>
> ... or improve the speed for overlays in Emacs :)
>
> -- 
>  Bastien

Re: Improving org-contacts performance (and state of development in general)

[TRS-80]

On 2020-09-06 22:52, Daryl Manning wrote:
> PS> As an outside feature though, interoperability of the org-contact
> formats with other operating system address books, most notable gnome
> contacts/evolution, goog contacts, and OSX address book would be high
> on my list in terms of improving org-contacts though. (eg, raw,
> structued info in all address books, and say perhaps notes or similar
> maintained and synced in ome manner.

Hi Daryl,

Good topic.

This is essentially the main reason I don't use org-contacts, even
though I have become a heavy Orgmode/Emacs user for almost everything
else over the past few years.

In my case, it's the Contacts on my Android phone, which seems to me
to be the primary place where I find myself adding/removing/editing my
contacts.  In my view, there /must/ be reliable and proper (/two-way/)
sync between the two.  Since that doesn't exist (that I am aware of)
my Android phone won out over org-contacts so far.

I haven't looked into it in a while, but at one point I thought that
using some common and open format would be the best idea.  Something
like CardDAV.

I actually had it working (syncing) with my NextCloud instance when I
had that up and running, but for me NextCloud proved to be to "heavy"
in general so I moved away from it, preferring instead lighter weight
tools to do the same things.  And so far I only implemented Syncthing
for the syncing part, and nothing for the CalDAV/CardDAV part (yet)
although I have had my eye on Radicale already for a while.

Maybe you know all this already, I'm just adding my $0.02.

Cheers,
TRS-80