Build agenda asynchronously

Build agenda asynchronously

[dalanicolai]

[-- Attachment #1: Type: text/plain, Size: 435 bytes --]

There was a question on Stack-Exchange if it would be possible to build
*org-agenda* asynchronously. I could not quickly find any good answers
neither in the mailing lists nor anywhere else.

As I assume this will be a quite helpful feature for some more users, I am
notifying you about it here.

The question with its answer can be found at
https://emacs.stackexchange.com/questions/68123/async-org-agenda/68136#68136

Cheers,
Daniel

[-- Attachment #2: Type: text/html, Size: 649 bytes --]

Re: Build agenda asynchronously

[Tim Cross]

Emacs' support for asynchronous operations is at best primitive. There
is built-in support for calling processes asynchonously and
there is some other development work to set the stage for adding threads,
but I think general asynchronous processing inside Emacs is a long way
off. A lot of how Emacs lisp works fundamentally lacks the low level
control structures necessary to make data structures and operations on
those structures thread safe. This means you have to work at a very low
level in order to ensure code is thread safe and that simply isn't
practical. Even defining the basic model for an asynchronous emacs lisp
is non-trivial and once you have the model, you ahve to implement it.

At this time, this is a non-starter. It would be a mistake to expect
asynchronous processing within Emacs lisp any time soon. Best we can
hope for now is asynchronous execution of external processes and even
that falls apart as soon as user input needs to be factored in. 

dalanicolai <dalanicolai@gmail.com> writes:

> There was a question on Stack-Exchange if it would be possible to build *org-agenda* asynchronously. I could not quickly find
> any good answers neither in the mailing lists nor anywhere else.
>
> As I assume this will be a quite helpful feature for some more users, I am notifying you about it here.
>
> The question with its answer can be found at
> https://emacs.stackexchange.com/questions/68123/async-org-agenda/68136#68136
>
> Cheers,
> Daniel

Re: Build agenda asynchronously

[Dr. Arne Babenhauserheide]

[-- Attachment #1: Type: text/plain, Size: 950 bytes --]


Tim Cross <theophilusx@gmail.com> writes:

> Emacs' support for asynchronous operations is at best primitive. There
> is built-in support for calling processes asynchonously and
> there is some other development work to set the stage for adding threads,
> but I think general asynchronous processing inside Emacs is a long way
> off. A lot of how Emacs lisp works fundamentally lacks the low level
> control structures necessary to make data structures and operations on
> those structures thread safe. This means you have to work at a very low
> level in order to ensure code is thread safe and that simply isn't
> practical. Even defining the basic model for an asynchronous emacs lisp
> is non-trivial and once you have the model, you ahve to implement it.

Maybe it could be possible to fire up a second Emacs and retrieve the
agenda-buffer?

Best wishes,
Arne
-- 
Unpolitisch sein
heißt politisch sein
ohne es zu merken

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 1125 bytes --]

Re: Build agenda asynchronously

[Tim Cross]

"Dr. Arne Babenhauserheide" <arne_bab@web.de> writes:

> [[PGP Signed Part:Undecided]]
>
> Tim Cross <theophilusx@gmail.com> writes:
>
>> Emacs' support for asynchronous operations is at best primitive. There
>> is built-in support for calling processes asynchonously and
>> there is some other development work to set the stage for adding threads,
>> but I think general asynchronous processing inside Emacs is a long way
>> off. A lot of how Emacs lisp works fundamentally lacks the low level
>> control structures necessary to make data structures and operations on
>> those structures thread safe. This means you have to work at a very low
>> level in order to ensure code is thread safe and that simply isn't
>> practical. Even defining the basic model for an asynchronous emacs lisp
>> is non-trivial and once you have the model, you ahve to implement it.
>
> Maybe it could be possible to fire up a second Emacs and retrieve the
> agenda-buffer?
>

Yes, I've seen people who have done things like that. They have their
'main' emacs instance running where they 'do stuff' and another instance
running where they read mail, check their agenda/calendar or run other
things which tend to 'block'.

There are some issues with this approach though. Some of which are not
obvious.

- Emacs can consume a fair amount of memory. Running multiple instances
  can use a fair amount of memory.
- Startup times make it less appealing as a 'run it up when you need it'
  solution.
- You need to have an understanding of the interaction with various
  cache and other disk files Emacs uses to ensure each instance doesn't
  tread on the toes of the other. 
- A 'strategy' with regards to reverting buffers to file contents.
  Consider adding a todo in your agenda while at the same time, working
  on a related org file in the other instance.  

Provided you know the rules, can operate in an understood disciplined
manner and have a system with sufficient resources, it certainly can be
a viable approach.

Personally, I took a different route. I keep the number of files which
contribute to my agenda to a minimum and have an easy way to
update/change that list. I can quickly switch agenda contexts depending
on what I'm doing. For example, when I'm at work, I'm not interested in
any of my 'home' tasks, events, etc. As a result, generation of various
agenda views is fast and I don't need to wait. I do have an 'everything'
context, so when I do need that, I can get a complete overview. It does
take longer to generate and will block, but I rarely need it. My
calendar.org file, which contains all my meetings is shared across all
views so that I avoid 'double booking'. I do tend to put all meetings in
Google calendar because it makes it easy to share them with other
devices (tablet, phone, laptop, other people), but I use my icsorg
script to import that into my calendar.org file. 

Re: Build agenda asynchronously

[Milan Zamazal]

>>>>> "TC" == Tim Cross <theophilusx@gmail.com> writes:

    TC> Personally, I took a different route. I keep the number of files
    TC> which contribute to my agenda to a minimum and have an easy way
    TC> to update/change that list. I can quickly switch agenda contexts
    TC> depending on what I'm doing. 

It’s always advisable to restrict agenda sources to what’s actually
relevant.  But besides that, all my problems with agenda slowness were
resolved once I started to use org-super-agenda
(https://github.com/alphapapa/org-super-agenda).  It’s both more
powerful and much faster than the standard Org agenda.

Regards,
Milan

Re: Build agenda asynchronously

[Tim Cross]

Milan Zamazal <pdm@zamazal.org> writes:

>>>>>> "TC" == Tim Cross <theophilusx@gmail.com> writes:
>
>     TC> Personally, I took a different route. I keep the number of files
>     TC> which contribute to my agenda to a minimum and have an easy way
>     TC> to update/change that list. I can quickly switch agenda contexts
>     TC> depending on what I'm doing. 
>
> It’s always advisable to restrict agenda sources to what’s actually
> relevant.  But besides that, all my problems with agenda slowness were
> resolved once I started to use org-super-agenda
> (https://github.com/alphapapa/org-super-agenda).  It’s both more
> powerful and much faster than the standard Org agenda.
>

Actually, that one has been on my todo list to cehck out for a while.
Might be time to bump the priority a bit!

Re: Build agenda asynchronously

[Henrik Frisk]

[-- Attachment #1: Type: text/plain, Size: 295 bytes --]

Den ons 18 aug. 2021 kl 02:17 skrev Tim Cross <theophilusx@gmail.com>:

>
> Milan Zamazal <pdm@zamazal.org> writes:
>
> Actually, that one has been on my todo list to cehck out for a while.
> Might be time to bump the priority a bit!
>
> I just started using it and it works really well so far!

[-- Attachment #2: Type: text/html, Size: 704 bytes --]

Re: Build agenda asynchronously

[Henrik Frisk]

[-- Attachment #1: Type: text/plain, Size: 1128 bytes --]

Den tis 17 aug. 2021 kl 08:54 skrev Dr. Arne Babenhauserheide <
arne_bab@web.de>:

>
> Tim Cross <theophilusx@gmail.com> writes:
>
> > Emacs' support for asynchronous operations is at best primitive. There
> > is built-in support for calling processes asynchonously and
> > there is some other development work to set the stage for adding threads,
> > but I think general asynchronous processing inside Emacs is a long way
> > off. A lot of how Emacs lisp works fundamentally lacks the low level
> > control structures necessary to make data structures and operations on
> > those structures thread safe. This means you have to work at a very low
> > level in order to ensure code is thread safe and that simply isn't
> > practical. Even defining the basic model for an asynchronous emacs lisp
> > is non-trivial and once you have the model, you ahve to implement it.
>
> Maybe it could be possible to fire up a second Emacs and retrieve the
> agenda-buffer?
>
> This is what I do for commands like exporting the agenda which can take a
while with the -nw flag, but that is of course unsatisfactory for most use
cases.

/Henrik

[-- Attachment #2: Type: text/html, Size: 1573 bytes --]

Re: Build agenda asynchronously

[Ihor Radchenko]

dalanicolai <dalanicolai@gmail.com> writes:

> There was a question on Stack-Exchange if it would be possible to build
> *org-agenda* asynchronously. I could not quickly find any good answers
> neither in the mailing lists nor anywhere else.

There was a proof-of-concept code building agenda using Emacs threads:
https://orgmode.org/list/87k172ot2m.fsf@yantar92-laptop.i-did-not-set--mail-host-address--so-tickle-me/

However, if your agenda is blocking Emacs it will only get worse over
time. Async will not be good enough eventually. I recommend using
org-ql. It should be enough unless your agenda it taking really a lot of
time to build (10s of seconds).

Best,
Ihor