ICS agenda export exceeds max-specpdl-size probably because of org-depend (org-edna same?)

ICS agenda export exceeds max-specpdl-size probably because of org-depend (org-edna same?)

[Karl Voit]

Hi all,


BACKGROUND:

I'm using org-depend since ages[1]. As a matter of fact, it was the
initial reason why I started with Org mode in the first place.

One of my many workflows[2] exports the agenda for the upcoming 60
days into an ICS file via a cron job during night:

: emacs-snapshot --batch --load /home/vk/.emacs.d/init.el --eval '(progn (my-export-agenda))'

my-export-agenda[6] basically consists of:
: (org-agenda-list nil nil 60)
: (org-agenda-write (concat my-user-emacs-directory "var/export/agenda-export.ics"))

This ICS file is then postprocessed and uploaded to my calendar server
so that my mobile phone calendar is able to display events and
appointments.


This workflow stopped working end of December.


ERRORS and INVESTIGATION:

Error message when this batch run of Emacs exits after 5 hours(!):
: Lisp nesting exceeds ‘max-lisp-eval-depth’

I updated my Org version[3] recently to get v9.3 from git maint. This
was Wed Dec 4 10:37:19 2019 +0100. So the issue with the ICS export
did not appear at the same time of the update.

I also get:
: [...]
: Preparing diary...done
: Preparing diary...done
: Position saved to mark ring, go back with ‘C-c &’. [1038 times]
: org-before-first-heading-p: Variable binding depth exceeds max-specpdl-size

I changed the value for the variables without changing the result:
: (setq max-specpdl-size 10000)
: (setq max-lisp-eval-depth 50000)

I even upgraded my Emacs version from 25 to 27[3]. No change.

I reduced my set of org-agenda-files and did some test runs: most
single org files resulted in an ICS export. Some single org files did
result in the error.

Reducing the export period from 60 to 7 days did not change the error.

I also get following as execution cancellation error:
: Before first headline at position 586 in buffer  *temp*-99684

When running the job, I see *lots* lines like[4]:
: Garbage collecting...
: Garbage collecting...done

To me, it's a data-related issue with my Org mode files so that others
most likely are not able to reproduce my issue at all.

Disabling org-depend results in a working ICS export that took much
less time. I'm not sure if org-depend is directly related or just
reduces "the complexity" such that the job is able to be finalized.


QUESTIONS:

I'm looking for ideas what I may check in order to be able to use
org-depend again.

Should I do a profiler report? How do I have to do this in order to
get good results?

Further more: is here somebody able to tell if org-edna[5] does have
this issues as well? After all, org-edna is somewhat more complex than
org-depend.


A migration from org-depend to org-edna would be doable to me but with
a decent effort. (I'm thinking of migrating only tasks that are not
DONE or CANCELLED as I do think closed tasks do not irritate org-edna
with old org-depend syntax.)



[1] https://karl-voit.at/2016/12/18/org-depend/
    Almost everything I use is :BLOCKER: properties and very
    rarely :TRIGGER: events.

[2] just a selection is described on https://karl-voit.at/2019/09/25/using-orgmode/

[3] Org mode version 9.3 (release_9.3-3-g46319b.dirty @ mixed installation!
    GNU Emacs 27.0.50 (build 1, x86_64-pc-linux-gnu, GTK+ Version
                       3.22.11) of 2019-12-15, unofficial emacs-snapshot build

[4] I setup up "gcmh" package because of a comment on:
    https://www.reddit.com/r/orgmode/comments/e9p84n/scaling_org_better_to_use_more_medsize_files_or/fcm5bsc/

[5] https://elpa.gnu.org/packages/org-edna.html

[6] https://github.com/novoid/dot-emacs/blob/master/config.org


-- 
get mail|git|SVN|photos|postings|SMS|phonecalls|RSS|CSV|XML into Org-mode:
       > get Memacs from https://github.com/novoid/Memacs <
Personal Information Management > http://Karl-Voit.at/tags/pim/
Emacs-related > http://Karl-Voit.at/tags/emacs/

Re: ICS agenda export exceeds max-specpdl-size probably because of org-depend (org-edna same?)

[Bastien]

Hi Karl,

did you manage to fix this one?

I read "Property drawers allowed before first headline" in
etc/ORG-NEWS from the master branch:

  Property drawers are now allowed before the first headline.

  Org mode is moving more towards making things before the first
  headline behave just as if it was at outline level 0.  Inheritance
  for properties will work also for this level.  In other words:
  defining things in a property drawer before the first headline will
  make them "inheritable" for all headlines.

I guess your problem might be related to this new behavior.

2 cents,

-- 
 Bastien

Re: ICS agenda export exceeds max-specpdl-size probably because of org-depend (org-edna same?)

[Karl Voit]

* Bastien <bzg@gnu.org> wrote:
> Hi Karl,

Hi Bastien,

> did you manage to fix this one?

Well ...

> I read "Property drawers allowed before first headline" in
> etc/ORG-NEWS from the master branch:
>
>   Property drawers are now allowed before the first headline.
>
>   Org mode is moving more towards making things before the first
>   headline behave just as if it was at outline level 0.  Inheritance
>   for properties will work also for this level.  In other words:
>   defining things in a property drawer before the first headline will
>   make them "inheritable" for all headlines.
>
> I guess your problem might be related to this new behavior.

Since I'm not using drawers before the first headline, I don't
understand how this might relate to my issue.

However, I just re-enabled org-depend which I had disabled for
batch mode only in order to get the export done. Then I ran the
batch process for the export and it finished without an error within
1min53sec. :-)

So the issue seems to be gone (for now) but I don't know why it
appeared in the first place nor why it disappeared in the meantime.

I'm using Org mode from maint. Currently on my side: d20e45b18 (3
days ago). The version that had the issue was from November or
December. I could provide the exact ID hash if anybody is
interested.

Thanks anyway and thanks for the pointer with the new feature. That
might come handy when there is a property which does not have a
file-local version.

-- 
get mail|git|SVN|photos|postings|SMS|phonecalls|RSS|CSV|XML into Org-mode:
       > get Memacs from https://github.com/novoid/Memacs <
Personal Information Management > http://Karl-Voit.at/tags/pim/
Emacs-related > http://Karl-Voit.at/tags/emacs/

Re: ICS agenda export exceeds max-specpdl-size probably because of org-depend (org-edna same?)

[Bastien]

Hi Karl,

Karl Voit <devnull@Karl-Voit.at> writes:

> So the issue seems to be gone (for now) but I don't know why it
> appeared in the first place nor why it disappeared in the meantime.

/me looks for a good xkcd quote. :)

-- 
 Bastien

Re: ICS agenda export exceeds max-specpdl-size probably because of org-depend (org-edna same?)

[Karl Voit]

* Bastien <bzg@gnu.org> wrote:
> Hi Karl,
>
> Karl Voit <devnull@Karl-Voit.at> writes:
>
>> So the issue seems to be gone (for now) but I don't know why it
>> appeared in the first place nor why it disappeared in the meantime.
>
> /me looks for a good xkcd quote. :)

Alternatively: https://en.wikipedia.org/wiki/Haiku_in_English  ;-)

-- 
get mail|git|SVN|photos|postings|SMS|phonecalls|RSS|CSV|XML into Org-mode:
       > get Memacs from https://github.com/novoid/Memacs <
Personal Information Management > http://Karl-Voit.at/tags/pim/
Emacs-related > http://Karl-Voit.at/tags/emacs/

Re: ICS agenda export exceeds max-specpdl-size probably because of org-depend (org-edna same?)

[Karl Voit]

Hi Bastien,

* Bastien <bzg@gnu.org> wrote:
> Hi Karl,
>
> did you manage to fix this one?
>
> I read "Property drawers allowed before first headline" in
> etc/ORG-NEWS from the master branch:
>
>   Property drawers are now allowed before the first headline.
>
>   Org mode is moving more towards making things before the first
>   headline behave just as if it was at outline level 0.  Inheritance
>   for properties will work also for this level.  In other words:
>   defining things in a property drawer before the first headline will
>   make them "inheritable" for all headlines.
>
> I guess your problem might be related to this new behavior.

A couple days ago, the issue re-appeared (with org-depend being
active again). To me, this is a clear indicator, that the issue is
content-related and not config/binary-related:

Emacs 26.3:

: emacs --batch --load /home/vk/.emacs.d/init.el --eval '(progn (my-export-agenda))'
: [...]
: Garbage collecting...
: Garbage collecting...done
: Garbage collecting...
: Garbage collecting...done
: Garbage collecting...
: Garbage collecting...done
: Position saved to mark ring, go back with ‘C-c &’.
: Position saved to mark ring, go back with ‘C-c &’.
: Position saved to mark ring, go back with ‘C-c &’.
: Position saved to mark ring, go back with ‘C-c &’.
: Position saved to mark ring, go back with ‘C-c &’.
: Position saved to mark ring, go back with ‘C-c &’.
: Garbage collecting...
: Garbage collecting...done
: Variable binding depth exceeds max-specpdl-size
: emacs --batch --load /home/vk/.emacs.d/init.el --eval   8029,80s user 4,30s system 99% cpu 2:14:02,16 total

Yes, that's over two hours.

With Emacs 27.0.50, the same job is currently running for over five hours. I
guess this will end with an error as well some day.

-- 
get mail|git|SVN|photos|postings|SMS|phonecalls|RSS|CSV|XML into Org-mode:
       > get Memacs from https://github.com/novoid/Memacs <
Personal Information Management > http://Karl-Voit.at/tags/pim/
Emacs-related > http://Karl-Voit.at/tags/emacs/

Re: ICS agenda export exceeds max-specpdl-size probably because of org-depend (org-edna same?)

[Bastien]

Hi Karl,

Karl Voit <devnull@Karl-Voit.at> writes:

> A couple days ago, the issue re-appeared (with org-depend being
> active again). To me, this is a clear indicator, that the issue is
> content-related and not config/binary-related:

Can you bisect and deduce what part of your contents is producing
this?  When you have a clue, could you also bisect and see what
change in Org introduced this problem for the problematic contents?

Such issues are difficult to reproduce and track outside your files.

Thanks!

-- 
 Bastien

Re: ICS agenda export exceeds max-specpdl-size probably because of org-depend (org-edna same?)

[Karl Voit]

Hi,

* Bastien <bzg@gnu.org> wrote:
>
> Karl Voit <devnull@Karl-Voit.at> writes:
>
>> A couple days ago, the issue re-appeared (with org-depend being
>> active again). To me, this is a clear indicator, that the issue is
>> content-related and not config/binary-related:
>
> Can you bisect and deduce what part of your contents is producing
> this?  

This is not trivial in my case since I'd have to merge all my Org
mode content into one single large file and make sure that I remove
content during bi-secting that has no link destination from the
remaining content. 

Personally, I'm not convinced that this method will lead to a
minimum example in any case. I can imagine that "number of Org mode
headings/lines" might be involved here which would not trigger the
error when bi-secting via removing.

Currently, I don't see that this is possible without half a day work
or (IMHO more likely) longer which I can't afford right now. :-(

> When you have a clue, could you also bisect and see what
> change in Org introduced this problem for the problematic contents?

This would be trivial when I could locate the issue in a minimal
example.

> Such issues are difficult to reproduce and track outside your files.

Totally agree. I thought somebody else might have faced the same
issue. As long as I am the only person affected, I don't assume that
it can be solved in a proper way. :-(

I ran another test with Emacs 27.0.50 and disabled org-depend. The
task finished after 4½ minutes without any error. Therefore,
following dependencies has to be the culprit here.

Moving from org-depend to org-edna is not trivial for me as well,
considering the amount of meta-data to be modified. On the other
hand, this would be interesting to see if org-edna can handle this
use-case in a better way. In case somebody has experience with
migration process and/or dependency lookup performance, I'm very
interested.

-- 
get mail|git|SVN|photos|postings|SMS|phonecalls|RSS|CSV|XML into Org-mode:
       > get Memacs from https://github.com/novoid/Memacs <
Personal Information Management > http://Karl-Voit.at/tags/pim/
Emacs-related > http://Karl-Voit.at/tags/emacs/

Re: ICS agenda export exceeds max-specpdl-size probably because of org-depend (org-edna same?)

[Bastien]

Hi Karl,

another suggestion: just send us a minimal extract of your files so
that we see how org-depend is called, together with what you do and
what goes wrong when you do it.  Perhaps this will lead to something.

I've never used org-edna, I'm curious to know if there are many users.

Thanks,

-- 
 Bastien

Re: ICS agenda export exceeds max-specpdl-size probably because of org-depend (org-edna same?)

[Karl Voit]

* Bastien <bzg@gnu.org> wrote:
> Hi Karl,

Hi,

> another suggestion: just send us a minimal extract of your files so
> that we see how org-depend is called, together with what you do and
> what goes wrong when you do it.  Perhaps this will lead to something.

Sure.

https://github.com/novoid/dot-emacs/blob/master/config.org contains:

| (setq my-user-emacs-directory "~/.emacs.d/")
| (use-package org-depend
|   :load-path  (lambda () (expand-file-name (concat my-user-emacs-directory "contrib/org-mode/contrib/lisp/")))
| )

... which loads org-depend from my maint Org mode (currently: 9.3
(release_9.3-102-gd20e45.dirty @ mixed installation![...]).

I only use org-depend for:

* TODO Example 1
:PROPERTIES:
:ID: 2020-02-20-foo
:END:

* TODO Example 2
:PROPERTIES:
:BLOCKER: 2020-02-20-foo
:END:

Very rarely, I am using:

* TODO Example 3
:PROPERTIES:
:TRIGGER: 2020-02-20-foo
:END:

Most of the time, source-destination are within the same file. However, in some
cases of using BLOCKER, the two headings might be in two different files.

That's basically it: mostly BLOCKER and a few TRIGGER. I'm unsure if there is
currently a TRIGGER that is still waiting to be activated.

(By the way: is there a built-in feature to look for IDs that are linked
somewhere (BLOCKER properties or ID links) that are broken?)

Ad "what goes wrong when you do it": in my daily work, org-depend is
working fine. The only thing that does not work is exporting my
agenda with:

| emacs --batch --load /home/vk/.emacs.d/init.el --eval '(progn (my-export-agenda))'

and:

|  (defun my-export-agenda()
|    "Exports Org-mode agenda to ics file"
|    (interactive)
|    (save-some-buffers)
|
|    ;; I don't want the error messages in my exported agenda:
|    (setq org-agenda-files (remove "~/org/errors.org" org-agenda-files))
|
|    (setq max-specpdl-size 10000) ;; does not solve issue
|    (setq max-lisp-eval-depth 50000) ;; does not solve issue
|
|    (org-agenda-list nil nil 60)
|    (org-agenda-write (concat my-user-emacs-directory "var/export/agenda-export.ics"))
|    )

With disabled org-depend, it is working and finishes in ~5 minutes. With
enabled org-depend it (sometimes) end up in an error as described. This can
take hours.

> I've never used org-edna, I'm curious to know if there are many users.

Me too ;-)

It's a bit more complicated in syntax but has advanced features org-depend does
not offer. The main reason why I'm thinking of switching is that org-edna
supports something like TRIGGER that also adds relative SCHEDULED date-stamps.
This allows for "if $THAT task gets done, mark $NEXT task as a TODO item and
add a SCHEDULED with three days in the future" which is very handy in some
cases.

> Thanks,

Oh, it's me, who is very thankful, considering the tricky situation to
investigate this issue.

-- 
get mail|git|SVN|photos|postings|SMS|phonecalls|RSS|CSV|XML into Org-mode:
       > get Memacs from https://github.com/novoid/Memacs <
Personal Information Management > http://Karl-Voit.at/tags/pim/
Emacs-related > http://Karl-Voit.at/tags/emacs/

Re: ICS agenda export exceeds max-specpdl-size probably because of org-depend (org-edna same?)

[Bastien]

Hi Karl,

Karl Voit <devnull@Karl-Voit.at> writes:

> With disabled org-depend, it is working and finishes in ~5 minutes. With
> enabled org-depend it (sometimes) end up in an error as described.

Do you need org-depend when exporting?

If not, I simply suggest to turn org-depend off... or to let-bind
`org-trigger-hook' and `org-blocker-hook' to nil in your function.

Probably org-depend does a lot of moving back and forth in files to
check for IDs, and that may explain the time spend by this command
when you export many files.

-- 
 Bastien

Re: ICS agenda export exceeds max-specpdl-size probably because of org-depend (org-edna same?)

[Karl Voit]

Bonjour Bastien,

* Bastien <bzg@gnu.org> wrote:
>
> Karl Voit <devnull@Karl-Voit.at> writes:
>
>> With disabled org-depend, it is working and finishes in ~5 minutes. With
>> enabled org-depend it (sometimes) end up in an error as described.
>
> Do you need org-depend when exporting?

Well: in all cases where I defined a dependency of task B to a task
A, with disabled org-depend I end up seeing task B on the agenda
although I can not do it at the moment.

So as long as I don't use dependencies, it does not matter. However,
I'm organizing a lecture in the summer term using a very large and
complicated yankpad/yasnippet template for the whole term that is
using many dependencies for approx. 70 todos. With disabled
org-depend, I end up seeing all 70 todos although I may be able to
work on two or three of them at any moment in time.

> If not, I simply suggest to turn org-depend off... or to let-bind
> `org-trigger-hook' and `org-blocker-hook' to nil in your function.
>
> Probably org-depend does a lot of moving back and forth in files to
> check for IDs, and that may explain the time spend by this command
> when you export many files.

This sounds reasonable.

I'm fine with - let's say - +50% 1-day agenda duration and I'm fine
with even an hour of 60d agenda export to ics with enabled
org-depend.  The show-stopper is, when org-depend does run in an
error after hours of exporting without a result. This way, I'm
losing my Org->Android/Radicale workflow or my dependencies.

The longer this thread gets, the more I have to accept that I might
be the only one using dependencies in Org mode. I'm puzzled here
because I do think that dependencies are such a natural thing to
want and use. (I also have several ideas to improve usability of
them to minimize manual effort when defining dependencies.)

So maybe I have to invest a weekend or so to bisect the issue. My
gut feeling tells me that I won't like the output somehow.

-- 
get mail|git|SVN|photos|postings|SMS|phonecalls|RSS|CSV|XML into Org-mode:
       > get Memacs from https://github.com/novoid/Memacs <
Personal Information Management > http://Karl-Voit.at/tags/pim/
Emacs-related > http://Karl-Voit.at/tags/emacs/

Re: ICS agenda export exceeds max-specpdl-size probably because of org-depend (org-edna same?)

[Bastien]

Hi Karl,

Karl Voit <devnull@Karl-Voit.at> writes:

> So maybe I have to invest a weekend or so to bisect the issue. My
> gut feeling tells me that I won't like the output somehow.

In the meantime, can you share the values of your org-id-* options?

Thanks,

-- 
 Bastien

Re: ICS agenda export exceeds max-specpdl-size probably because of org-depend (org-edna same?)

[Karl Voit]

* Bastien <bzg@gnu.org> wrote:
> Hi Karl,
>
> Karl Voit <devnull@Karl-Voit.at> writes:
>
>> So maybe I have to invest a weekend or so to bisect the issue. My
>> gut feeling tells me that I won't like the output somehow.
>
> In the meantime, can you share the values of your org-id-* options?

Easy: I could not find any change of the defaults in my
configuration[1]. So all of them have to be the default values.

[1] https://github.com/novoid/dot-emacs/blob/master/config.org

-- 
get mail|git|SVN|photos|postings|SMS|phonecalls|RSS|CSV|XML into Org-mode:
       > get Memacs from https://github.com/novoid/Memacs <
Personal Information Management > http://Karl-Voit.at/tags/pim/
Emacs-related > http://Karl-Voit.at/tags/emacs/

Re: ICS agenda export exceeds max-specpdl-size probably because of org-depend (org-edna same?)

[Bastien]

Hi Karl,

Karl Voit <devnull@Karl-Voit.at> writes:

> Easy: I could not find any change of the defaults in my
> configuration[1]. So all of them have to be the default values.
>
> [1] https://github.com/novoid/dot-emacs/blob/master/config.org

I would try setting `org-agenda-before-write-hook' to nil (it is set
to `org-agenda-add-entry-text' by default.

Also, I see `org-icalendar-include-body' is set to 1000 in your setup.

That's a lot and this will generate a lot of strings replacements,
I suggest trying a lower value for your export-and-write function.

HTH,

-- 
 Bastien

Re: ICS agenda export exceeds max-specpdl-size probably because of org-depend (org-edna same?)

[Karl Voit]

Hi Bastien,

* Bastien <bzg@gnu.org> wrote:
>
> Karl Voit <devnull@Karl-Voit.at> writes:
>
>> Easy: I could not find any change of the defaults in my
>> configuration[1]. So all of them have to be the default values.
>>
>> [1] https://github.com/novoid/dot-emacs/blob/master/config.org
>
> I would try setting `org-agenda-before-write-hook' to nil (it is set
> to `org-agenda-add-entry-text' by default.
>
> Also, I see `org-icalendar-include-body' is set to 1000 in your setup.
>
> That's a lot and this will generate a lot of strings replacements,
> I suggest trying a lower value for your export-and-write function.

Thanks for the advice. I ran three test runs in order to find out how output is
affected:


Testrun with disabled org-depend:
emacs-snapshot --batch --load /home/vk/.emacs.d/init.el --eval '(progn (my-export-agenda))
   261.95s user 1.57s system 99% cpu 4:24.01 total

→ I got a result without error


Testrun with enabled org-depend:
emacs-snapshot --batch --load /home/vk/.emacs.d/init.el --eval '(progn (my-export-agenda))'
   117.58s user 1.46s system 99% cpu 1:59.60 total

→ much faster result without error :-)  (probably caching?)


revoking the two suggested changes and do another test run with org-depend, expecting error:
emacs-snapshot --batch --load /home/vk/.emacs.d/init.el --eval '(progn (my-export-agenda))'
   114.38s user 1.53s system 99% cpu 1:56.33 total

→ much faster result without error (contrary to expection)


→ the two setting changes do not have an effect for now regarding
  error/no error due to the fact that I can not reproduce the
  error.

Retrying when error re-appears.

-- 
get mail|git|SVN|photos|postings|SMS|phonecalls|RSS|CSV|XML into Org-mode:
       > get Memacs from https://github.com/novoid/Memacs <
Personal Information Management > http://Karl-Voit.at/tags/pim/
Emacs-related > http://Karl-Voit.at/tags/emacs/

Re: ICS agenda export exceeds max-specpdl-size probably because of org-depend (org-edna same?)

[Bastien]

Hi Karl,

Karl Voit <devnull@Karl-Voit.at> writes:

> Retrying when error re-appears.

OK, thanks for the feedback!

-- 
 Bastien

Re: ICS agenda export exceeds max-specpdl-size probably because of org-depend (org-edna same?)

[Karl Voit]

This week, the error re-appeared.

* Karl Voit <devnull@Karl-Voit.at> wrote:
>
> Thanks for the advice. I ran three test runs in order to find out how output is
> affected:
>
>
> Testrun with disabled org-depend:
> emacs-snapshot --batch --load /home/vk/.emacs.d/init.el --eval '(progn (my-export-agenda))
>    261.95s user 1.57s system 99% cpu 4:24.01 total
>
> → I got a result without error
>
>
> Testrun with enabled org-depend:
> emacs-snapshot --batch --load /home/vk/.emacs.d/init.el --eval '(progn (my-export-agenda))'
>    117.58s user 1.46s system 99% cpu 1:59.60 total
>
> → much faster result without error :-)  (probably caching?)
>
>
> revoking the two suggested changes and do another test run with org-depend, expecting error:
> emacs-snapshot --batch --load /home/vk/.emacs.d/init.el --eval '(progn (my-export-agenda))'
>    114.38s user 1.53s system 99% cpu 1:56.33 total
>
> → much faster result without error (contrary to expection)
>
>
> → the two setting changes do not have an effect for now regarding
>   error/no error due to the fact that I can not reproduce the
>   error.
>
> Retrying when error re-appears.

- setting `org-agenda-before-write-hook' to nil
- `org-icalendar-include-body': 42
- enabled org-depend

: emacs-snapshot --batch --load /home/vk/.emacs.d/init.el --eval '(progn (my-export-agenda))'

→ did not finish after running for 4 hours (aborted)


- setting `org-agenda-before-write-hook' to nil
- `org-icalendar-include-body': 42
- *disabled org-depend*

→ produced output within 4min39s


- setting `org-agenda-before-write-hook' to default
- `org-icalendar-include-body': to default
- disabled org-depend

→ produced output within 2min07s


Any other test you want me to run?

-- 
get mail|git|SVN|photos|postings|SMS|phonecalls|RSS|CSV|XML into Org-mode:
       > get Memacs from https://github.com/novoid/Memacs <
Personal Information Management > http://Karl-Voit.at/tags/pim/
Emacs-related > http://Karl-Voit.at/tags/emacs/