Is there a size limit to org files?

Is there a size limit to org files?

[Carlo Tambuatco]

[-- Attachment #1: Type: text/plain, Size: 1016 bytes --]

I noticed that an org file I was using for my capture templates was getting
quite large
and that my capture templates that were pasting TODO entries in that file
were starting to exhibit
strange behavior. Mostly just getting stuck at a mysterious "clipboard
pasted to level 2 subtree"
message, then it would just hang there until I hit ctrl g.

Since this file was getting quite big and unwieldy, > 5000 lines, I decided
to split it up, so that
the various entries under their headers would be split up into different
files. I basically just
put each header into its own separate file, and updated my
org-capture-templates accordingly.

Since then, no strange behavior, no stalling and hanging for no apparent
reason.

So, I was wondering if there is hardcoded somewhere in org some kind of
size limit to the
files it has to deal with, or is this some limitation arising naturally
from the size of
the clipboard? I'm not sure how org handles large files when considering
how to paste
entries from capture templates.

[-- Attachment #2: Type: text/html, Size: 1224 bytes --]

Re: Is there a size limit to org files?

[Ihor Radchenko]

Carlo Tambuatco <oraclmaster@gmail.com> writes:

> I noticed that an org file I was using for my capture templates was getting
> quite large
> and that my capture templates that were pasting TODO entries in that file
> were starting to exhibit
> strange behavior. Mostly just getting stuck at a mysterious "clipboard
> pasted to level 2 subtree"
> message, then it would just hang there until I hit ctrl g.

That could be a bug, but it is impossible to fix it without knowing more
details.

> Since this file was getting quite big and unwieldy, > 5000 lines, I
> decided ...

I am using 100x (yes, 100 times) larger org file right now. With 500k lines.


> So, I was wondering if there is hardcoded somewhere in org some kind of
> size limit to the
> files it has to deal with, or is this some limitation arising naturally
> from the size of
> the clipboard? I'm not sure how org handles large files when considering
> how to paste
> entries from capture templates.

There is no size limit on Org files. Although certain operations may
become slower when the file size reaches tens of megabytes.

Clipboard should normally not be a problem either.

If you can interrupt with C-g, there might be some infinite loop
somewhere in the code.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Is there a size limit to org files?

[Carlo Tambuatco]

[-- Attachment #1: Type: text/plain, Size: 2047 bytes --]

You might want to look into what changed between org 9.5.5 and 9.6, because
that is when this strange behavior started (for me anyway). I've been
hanging onto
9.5.5 for as long as I could until I upgraded to emacs 29. Then the above
mentioned
strange behavior resumed. My fix for it was to split up my large org files
into
smaller files, and so far no strange behavior.



On Fri, Sep 8, 2023 at 12:51 PM Ihor Radchenko <yantar92@posteo.net> wrote:

> Carlo Tambuatco <oraclmaster@gmail.com> writes:
>
> > I noticed that an org file I was using for my capture templates was
> getting
> > quite large
> > and that my capture templates that were pasting TODO entries in that file
> > were starting to exhibit
> > strange behavior. Mostly just getting stuck at a mysterious "clipboard
> > pasted to level 2 subtree"
> > message, then it would just hang there until I hit ctrl g.
>
> That could be a bug, but it is impossible to fix it without knowing more
> details.
>
> > Since this file was getting quite big and unwieldy, > 5000 lines, I
> > decided ...
>
> I am using 100x (yes, 100 times) larger org file right now. With 500k
> lines.
>
>
> > So, I was wondering if there is hardcoded somewhere in org some kind of
> > size limit to the
> > files it has to deal with, or is this some limitation arising naturally
> > from the size of
> > the clipboard? I'm not sure how org handles large files when considering
> > how to paste
> > entries from capture templates.
>
> There is no size limit on Org files. Although certain operations may
> become slower when the file size reaches tens of megabytes.
>
> Clipboard should normally not be a problem either.
>
> If you can interrupt with C-g, there might be some infinite loop
> somewhere in the code.
>
> --
> Ihor Radchenko // yantar92,
> Org mode contributor,
> Learn more about Org mode at <https://orgmode.org/>.
> Support Org development at <https://liberapay.com/org-mode>,
> or support my work at <https://liberapay.com/yantar92>
>

[-- Attachment #2: Type: text/html, Size: 2851 bytes --]

Re: Is there a size limit to org files?

[Ihor Radchenko]

Carlo Tambuatco <oraclmaster@gmail.com> writes:

> You might want to look into what changed between org 9.5.5 and 9.6, because
> that is when this strange behavior started (for me anyway)...

Unfortunately, such information is not helpful. A lot have changed.

If you observe the problem in future, I encourage you to follow the
steps described in https://orgmode.org/manual/Feedback.html#Feedback, so
that we can identify and fix the bug.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Is there a size limit to org files?

[Max Nikulin]

On 08/09/2023 23:52, Ihor Radchenko wrote:
> Carlo Tambuatco writes:
> 
>> I noticed that an org file I was using for my capture templates was getting
>> quite large
>> and that my capture templates that were pasting TODO entries in that file
>> were starting to exhibit
>> strange behavior. Mostly just getting stuck at a mysterious "clipboard
>> pasted to level 2 subtree"
>> message, then it would just hang there until I hit ctrl g.
> 
> That could be a bug, but it is impossible to fix it without knowing more
> details.

Looks like a follow-up of
https://list.orgmode.org/CAJb92ExdvB7FMkSnmZAFTs4AuSZHBp5Rj4bK-4EWAGAbAGgC=g@mail.gmail.com/
Carlo Tambuatco. [BUG] Org-9.6.x org templates hang on 'clipboard pasted 
as level 2 subtree'... Mon, 3 Apr 2023 15:53:22 -0400

Certainly a way to reproduce the issue or detailed enough stack traces 
are required.

Have you tried to run `org-lint'?

I have seen a couple of times emacs consuming 100% CPU and not reacting 
to C-g, but in that cases I was not ready to debug these issues. Almost 
certainly it was unrelated to capture a clipboard, more likely 
incremental search or a wrong key while calling it.

Re: Is there a size limit to org files?

[Marcin Borkowski]

On 2023-09-08, at 18:26, Carlo Tambuatco <oraclmaster@gmail.com> wrote:

> I noticed that an org file I was using for my capture templates was getting
> quite large
> and that my capture templates that were pasting TODO entries in that file
> were starting to exhibit
> strange behavior. Mostly just getting stuck at a mysterious "clipboard
> pasted to level 2 subtree"
> message, then it would just hang there until I hit ctrl g.
>
> Since this file was getting quite big and unwieldy, > 5000 lines, I decided
> to split it up, so that
> the various entries under their headers would be split up into different
> files. I basically just
> put each header into its own separate file, and updated my
> org-capture-templates accordingly.
>
> Since then, no strange behavior, no stalling and hanging for no apparent
> reason.
>
> So, I was wondering if there is hardcoded somewhere in org some kind of
> size limit to the
> files it has to deal with, or is this some limitation arising naturally
> from the size of
> the clipboard? I'm not sure how org handles large files when considering
> how to paste
> entries from capture templates.

It's a /very/ long shot, but I noticed that editing large tables (1000+
lines) can become pretty slow.  Apart from that, I have Org files with
tens of thousands lines and they work pretty well.

Best,

-- 
Marcin Borkowski
http://mbork.pl

Re: Is there a size limit to org files?

[Ihor Radchenko]

Marcin Borkowski <mbork@mbork.pl> writes:

> It's a /very/ long shot, but I noticed that editing large tables (1000+
> lines) can become pretty slow.  Apart from that, I have Org files with
> tens of thousands lines and they work pretty well.

Expected. The current code re-inserts the whole table every time you
need to re-align it. You can guess how such approach scales.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Is there a size limit to org files?

[Marcin Borkowski]

On 2023-09-08, at 19:37, Ihor Radchenko <yantar92@posteo.net> wrote:

> Marcin Borkowski <mbork@mbork.pl> writes:
>
>> It's a /very/ long shot, but I noticed that editing large tables (1000+
>> lines) can become pretty slow.  Apart from that, I have Org files with
>> tens of thousands lines and they work pretty well.
>
> Expected. The current code re-inserts the whole table every time you
> need to re-align it. You can guess how such approach scales.

Agreed, just wanted to point out one possible reason for slowness.

Another one, it seems, is clocking in when the entry already has a lot
of clock entries.  I have a headline with 8000+ clock entries and
starting another one takes about half a second.

Best,

-- 
Marcin Borkowski
http://mbork.pl

Re: Is there a size limit to org files?

[Ihor Radchenko]

Marcin Borkowski <mbork@mbork.pl> writes:

> Another one, it seems, is clocking in when the entry already has a lot
> of clock entries.  I have a headline with 8000+ clock entries and
> starting another one takes about half a second.

Yup. The list parser is not incremental (for historic reasons), so every
time you change a list, it is re-parsed from scratch.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>