[Bug] org-update-dblock randomly broken for more than 10 columns and larger files

[Bug] org-update-dblock randomly broken for more than 10 columns and larger files

[Jakob Schöttl]

[-- Attachment #1: Type: text/plain, Size: 1519 bytes --]


I have an org file, about 3k lines with nested headings and property 
drawers.

org-update-dblock on this line inserts a table but only yields a subset 
of the actual headings:

#+BEGIN: columnview :hlines 1 :id bc3cd73e-bb35-46ac-8ab3-3e8bc33be9d1 
:maxlevel 3 :format "%50ITEM(Box) %TODO %TAGS %batch %hwversion 
%ausgeliefert %reinigungsmittel %evaporator_sensor_defekt 
%neuer_innenlüfter %med_netzteil %os"

Whereas this line works:

#+BEGIN: columnview :hlines 1 :id bc3cd73e-bb35-46ac-8ab3-3e8bc33be9d1 
:maxlevel 3 :format "%50ITEM(Box) %TODO %TAGS %batch %hwversion 
%ausgeliefert %reinigungsmittel %evaporator_sensor_defekt 
%neuer_innenlüfter %med_netzteil"

The only difference is that the first version includes 11 (or more) 
fields and the second version only includes 10 fields.

Also, when I repeat org-update-dblock, it yields different results every 
other time and the table is slowly growing, see attached screenshots. I 
also noticed that org-update-dblock seems to randomly add trailing 
whitespaces to headings or lines like ":END:"

The structure of the org file is attached. I tried for two hours to 
provide a minimal example org file but – like randomly – when I shorten 
the file radically or remove fields from the colmunview it works. I 
can't figure out what exactly makes org mode struggling.

Even if I delete all content in the file where lines do not start with 
regex /^\*/ or /^:[a-zA-Z]/, the inserted table is incomplete.

Can someone help to debug this further?

- Jakob


[-- Attachment #2: structure.org --]
[-- Type: text/org, Size: 527 bytes --]

* Table

https://orgmode.org/manual/Capturing-column-view.html

#+BEGIN: columnview :hlines 1 :id bc3cd73e-bb35-46ac-8ab3-3e8bc33be9d1 :maxlevel 3 :format "%50ITEM(Box) %TODO %TAGS %batch %hwversion %ausgeliefert %reinigungsmittel %evaporator_sensor_defekt %neuer_innenlüfter %med_netzteil %kein_scd30_adapter %os %sd_karte_getauscht"
#+END:

* Customers and products
:PROPERTIES:
:ID: bc3cd73e-bb35-46ac-8ab3-3e8bc33be9d1
:END:
**  <customer>
*** <product>
:PROPERTIES:
:prop1: val
[...]
:END:
**** <some sub headings>
[...]

[-- Attachment #3: 1.png --]
[-- Type: image/png, Size: 168387 bytes --]

[-- Attachment #4: 2.png --]
[-- Type: image/png, Size: 240839 bytes --]

[-- Attachment #5: 4.png --]
[-- Type: image/png, Size: 372241 bytes --]

[-- Attachment #6: 3.png --]
[-- Type: image/png, Size: 351619 bytes --]

Re: [Bug] org-update-dblock randomly broken for more than 10 columns and larger files

[Ihor Radchenko]

Jakob Schöttl <jschoett@gmail.com> writes:

> The structure of the org file is attached. I tried for two hours to 
> provide a minimal example org file but – like randomly – when I shorten 
> the file radically or remove fields from the colmunview it works. I 
> can't figure out what exactly makes org mode struggling.
>
> Even if I delete all content in the file where lines do not start with 
> regex /^\*/ or /^:[a-zA-Z]/, the inserted table is incomplete.
>
> Can someone help to debug this further?

It is hard to help much here without more details.
You may have to debug `org-dblock-write:columnview'.

What is your version of Org? Have you tried the latest dev version?
Latest release? 

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: [Bug] org-update-dblock randomly broken for more than 10 columns and larger files

[Jakob Schöttl]

Am 01.08.23 um 13:27 Ihor Radchenko:
> Jakob Schöttl <jschoett@gmail.com> writes:
>
>> The structure of the org file is attached. I tried for two hours to
>> provide a minimal example org file but – like randomly – when I shorten
>> the file radically or remove fields from the colmunview it works. I
>> can't figure out what exactly makes org mode struggling.
>>
>> Even if I delete all content in the file where lines do not start with
>> regex /^\*/ or /^:[a-zA-Z]/, the inserted table is incomplete.
>>
>> Can someone help to debug this further?
> It is hard to help much here without more details.
> You may have to debug `org-dblock-write:columnview'.
>
> What is your version of Org? Have you tried the latest dev version?
> Latest release?
>
Thanks Ihor for your reply. I just checked, on NixOS latest stable, I 
still have emacs 28.2 and org mode 9.5.

I also noticed that org-dblock-write:columnview modifies the buffer and 
adds a lot of trailing whitespace to property lines, headings, and also 
normal lines.

I'll try to reproduce it on latest org mode...

- Jakob

Re: [Bug] org-update-dblock randomly broken for more than 10 columns and larger files

[Jakob Schöttl]

So, org-update-dblock or org-dblock-write:columnview is adding trailing 
spaces in the org file. These spaces change the behavior of subsequent 
calls which will add even more spaces.

Here is a minimal example:

* Table
#+BEGIN: columnview :format "%a %b %c %d %e %f %g"
#+END:

This works as expected. But for each field you add, one trailing space 
is appended to the heading when the org-dblock-update is called.

In larger files, I cannot see any logic behind the added spaces so far.

I tested in Emacs 29.1 and org-mode 9.6.6. (Don't know how to use latest 
version from git repo.)

The function org-dblock-write:columnview is mostly by Nicolas Goaziou 
(from 2016 and 2018). Hello Nicolas, I've added you here. Maybe you have 
an idea what this could be?

Anyway, I'm afraid I can't contribute and fix it. I have no experience 
with elisp, tooling, debugging, and the org-mode code base. I need this 
export feature and will write a org2csv extraction script in Haskell. 
I'm aware that it would be much more helpful in org-mode itself – but 
sorry...

Re: [Bug] org-update-dblock randomly broken for more than 10 columns and larger files

[Ihor Radchenko]

Jakob Schöttl <jschoett@gmail.com> writes:

> So, org-update-dblock or org-dblock-write:columnview is adding trailing 
> spaces in the org file. These spaces change the behavior of subsequent 
> calls which will add even more spaces.

Confirmed.
Unimportant.

The internal implementation details of colview demand heading text to
have at least the number of characters equal to the number of fields. If
there are less, spaces are added. But not beyond the necessary number of
spaces.

This is not easy to change.
And I am not sure if it is necessarily a bug, though I can see how it
can be slightly annoying.

If someone reading this is bothered about the current behaviour, feel
free to work on the fix.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: [Bug] org-update-dblock randomly broken for more than 10 columns and larger files

[Jakob Schöttl]

Am 20.08.23 um 08:56 schrieb Ihor Radchenko:
> Jakob Schöttl <jschoett@gmail.com> writes:
>
>> So, org-update-dblock or org-dblock-write:columnview is adding trailing
>> spaces in the org file. These spaces change the behavior of subsequent
>> calls which will add even more spaces.
> Confirmed.
> Unimportant.
>
> The internal implementation details of colview demand heading text to
> have at least the number of characters equal to the number of fields. If
> there are less, spaces are added. But not beyond the necessary number of
> spaces.
>
> This is not easy to change.
> And I am not sure if it is necessarily a bug, though I can see how it
> can be slightly annoying.

The thing is, that this is only the smallest part of the bug. Spaces are 
not only added to headings but also to normal text lines and to ":END:" 
lines.

The spaces then lead to different table data in the dynamic block. I'll 
see if I can provide an example where it messes up the table data.

Re: [Bug] org-update-dblock randomly broken for more than 10 columns and larger files

[Jakob Schöttl]

Am 20.08.23 um 08:56 schrieb Ihor Radchenko:
> Jakob Schöttl <jschoett@gmail.com> writes:
>
>> So, org-update-dblock or org-dblock-write:columnview is adding trailing
>> spaces in the org file. These spaces change the behavior of subsequent
>> calls which will add even more spaces.
> Confirmed.
> Unimportant.

Got one:

* Table
#+BEGIN: columnview :format "%a %b %c %d %e %f %g %h"
#+END:
** foo
:PROPERTIES:
:a: a
:b: b
:c: c
:d: d
:e: e
:f: f
:g: g
:h: h
:END:


The first call to org-dblock-update adds some spaces and only has one 
empty line of data in the table.

The second call adds more spaces and has two lines of data in the table.

If %h is removed from :format, it works on the first run.

For larger files it's totally unreliable; I'd consider the columnview 
feature as broken.

Re: [Bug] org-update-dblock randomly broken for more than 10 columns and larger files

[Ihor Radchenko]

Jakob Schöttl <jschoett@gmail.com> writes:

> * Table
> #+BEGIN: columnview :format "%a %b %c %d %e %f %g %h"
> #+END:
> ** foo
> :PROPERTIES:
> :a: a
> :b: b
> :c: c
> :d: d
> :e: e
> :f: f
> :g: g
> :h: h
> :END:
>
> The first call to org-dblock-update adds some spaces and only has one 
> empty line of data in the table.
>
> The second call adds more spaces and has two lines of data in the table.

Thanks for the reproducer!
I committed a fix onto main.
https://git.savannah.gnu.org/cgit/emacs/org-mode.git/commit/?id=58c5c5882

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: [Bug] org-update-dblock randomly broken for more than 10 columns and larger files

[Jakob Schöttl]

Am 20.08.23 um 10:57 schrieb Ihor Radchenko:
> Thanks for the reproducer!
> I committed a fix onto main.
> https://git.savannah.gnu.org/cgit/emacs/org-mode.git/commit/?id=58c5c5882
Nice, thank you very much! Now, spaces are only added to headings and 
the resulting table is reproducible and correct.

Re: [Bug] org-update-dblock randomly broken for more than 10 columns and larger files

[Jakob Schöttl]

[-- Attachment #1: Type: text/plain, Size: 341 bytes --]

> Thanks for the reproducer!
> I committed a fix onto main.
> https://git.savannah.gnu.org/cgit/emacs/org-mode.git/commit/?id=58c5c5882

Hi Ihor, do you know anything on the merge progress of your fix into emacs?

I just tested with GNU Emacs 29.4 with --no-init-file and the minimal 
example still doesn't produce a correct table.

- Jakob

[-- Attachment #2: Type: text/html, Size: 956 bytes --]

Re: [Bug] org-update-dblock randomly broken for more than 10 columns and larger files

[Ihor Radchenko]

Jakob Schöttl <jschoett@gmail.com> writes:

>> Thanks for the reproducer!
>> I committed a fix onto main.
>> https://git.savannah.gnu.org/cgit/emacs/org-mode.git/commit/?id=58c5c5882
>
> Hi Ihor, do you know anything on the merge progress of your fix into emacs?
>
> I just tested with GNU Emacs 29.4 with --no-init-file and the minimal 
> example still doesn't produce a correct table.

May you please elaborate what exactly is not correct in the table?
I am not seeing any problem - no extra empty lines are added by the
second and subsequent calls.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: [Bug] org-update-dblock randomly broken for more than 10 columns and larger files

[Ihor Radchenko]

Ihor Radchenko <yantar92@posteo.net> writes:

>> I just tested with GNU Emacs 29.4 with --no-init-file and the minimal 
>> example still doesn't produce a correct table.
>
> May you please elaborate what exactly is not correct in the table?
> I am not seeing any problem - no extra empty lines are added by the
> second and subsequent calls.

Never mind my question.
Emacs 29.4 does not include Org 9.7, where the bug has been fixed.
You need either upcoming Emacs 30 or upgrade Org mode from ELPA.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: [Bug] org-update-dblock randomly broken for more than 10 columns and larger files

[Jakob Schöttl]

[-- Attachment #1: Type: text/plain, Size: 188 bytes --]

Am 12.07.24 um 13:56 schrieb Ihor Radchenko:
> Emacs 29.4 does not include Org 9.7, where the bug has been fixed.
> You need either upcoming Emacs 30 or upgrade Org mode from ELPA.
Thanks!

[-- Attachment #2: Type: text/html, Size: 564 bytes --]