[BUG] Incorrect indentation when there are invisible/diplay properties on the line [9.6.7 ( @ /home/huzf/.cache/emacs_configs/default/elpa.28/org-9.6.7/)]

[BUG] Incorrect indentation when there are invisible/diplay properties on the line [9.6.7 ( @ /home/huzf/.cache/emacs_configs/default/elpa.28/org-9.6.7/)]

[Hammer Hu]

--text follows this line--

Remember to cover the basics, that is, what you expected to happen and
what in fact did happen.  You don't know how to make a good report?  See

     https://orgmode.org/manual/Feedback.html#Feedback

Your bug report will be posted to the Org mailing list.
------------------------------------------------------------------------
Hi there,

I found this problem when indenting a region with `org-adapt-indentation=t'
and `org-modern-mode' is turned on. I initially posted this issue on the
issue tracker of `org-modern'
[[https://github.com/minad/org-modern/issues/144]], but as the author
suggested, I need to turn to the `org' side. Below is the problem.

When `org-modern' is installed and `org-modern-mode' is enable in a
buffer, the fontification is augmented. See
[[https://raw.githubusercontent.com/minad/org-modern/screenshots/example.gif]]
for a demonstration.  But when I have `org-modern-mode' turned on,
`org-adapt-indentation' to be `t', and indent some region, it becomes:

#+begin_quote
** A level-2 heading
   something

   something

            #+begin_quote
a block
            #+end_quote

and something
#+end_quote

While it should become:

#+begin_quote
** A level-2 heading
   something

   something

   #+begin_quote
   a block
   #+end_quote

   and something
#+end_quote

Here is the reply from minad (author of `org-modern'):

#+begin_quote
Thanks. I can reproduce the problem. It seems that `org-indent-region'
or `org-indent-line' has issues computing the indentation if there are
invisible/diplay properties on the line. This is not something we can
fix in org-modern, since `org-modern' only augments the fontification
and does not override indentation functionality. It should be repaired
in Org directly if even - maybe ask on the Org issue tracker? I suggest
you either don't use `org-adapt-identation' or set
`org-modern-block-name' to nil. With `org-modern-block-name=nil' the
issue also seems to go away.
#+end_quote

Best
-----

Emacs  : GNU Emacs 28.2.50 (build 1, x86_64-pc-linux-gnu, GTK+ Version 3.24.36, cairo version 1.17.6)
 of 2023-01-07
Package: Org mode version 9.6.7 ( @ /home/huzf/.cache/emacs_configs/default/elpa.28/org-9.6.7/)

current state:
==============
(setq
 org-archive-location "/home/huzf/org/personal/archive.org::* Archive"
 org-link-elisp-confirm-function 'yes-or-no-p
 org-bibtex-headline-format-function #[257 "\300\x01\236A\207" [:title] 3 "\n\n(fn ENTRY)"]
 org-log-done t
 org-startup-folded 'content
 org-agenda-files '("/home/huzf/org/personal/todo.org"
                    "/home/huzf/org/personal/inbox.org"
                    "/home/huzf/org/personal/archive.org")
 org-persist-after-read-hook '(org-element--cache-persist-after-read)
 org-refile-targets '((nil :maxlevel . 5) (org-agenda-files :maxlevel . 5))
 org-export-before-parsing-hook '(org-attach-expand-links)
 org-cycle-tab-first-hook '(org-babel-hide-result-toggle-maybe
                            org-babel-header-arg-expand)
 org-default-notes-file "/home/huzf/org/personal/inbox.org"
 org-refile-use-outline-path 'file
 org-archive-hook '(org-attach-archive-delete-maybe)
 org-cycle-hook '(org-cycle-hide-archived-subtrees org-cycle-show-empty-lines
                  org-cycle-optimize-window-after-visibility-change
                  org-cycle-display-inline-images)
 org-persist-before-read-hook '(org-element--cache-persist-before-read)
 org-mode-hook '(#<subr F616e6f6e796d6f75732d6c616d626461_anonymous_lambda_4>
                 #[0 "\300\301\302\303\304$\207"
                   [add-hook change-major-mode-hook org-fold-show-all append
                    local]
                   5]
                 #[0 "\300\301\302\303\304$\207"
                   [add-hook change-major-mode-hook org-babel-show-result-all
                    append local]
                   5]
                 org-babel-result-hide-spec org-babel-hide-all-hashes
                 org-modern-mode)
 org-agenda-clockreport-parameter-plist '(:link t :maxlevel 3)
 org-confirm-shell-link-function 'yes-or-no-p
 org-reveal-start-hook '(org-decrypt-entry)
 org-adapt-indentation t
 outline-isearch-open-invisible-function 'outline-isearch-open-invisible
 org-agenda-mode-hook '((closure (t) nil
                         (add-hook 'window-configuration-change-hook
                          'org-agenda-align-tags nil t)
                         )
                        )
 org-agenda-before-write-hook '(org-agenda-add-entry-text)
 org-agenda-after-show-hook '(org-show-entry)
 org-edit-timestamp-down-means-later t
 org-src-mode-hook '(org-src-babel-configure-edit-buffer
                     org-src-mode-configure-edit-buffer)
 org-confirm-elisp-link-function 'yes-or-no-p
 org-agenda-window-setup 'current-window
 org-tags-exclude-from-inheritance '("crypt")
 org-hide-leading-stars t
 org-todo-keywords '((sequence "TODO(t)" "NEXT(n)" "|" "DONE(d!/!)")
                     (sequence "PROJECT(p)" "|" "DONE(d!/!)" "CANCELLED(c@/!)")
                     (sequence "WAITING(w@/!)" "DELEGATED(e!)" "HOLD(h)" "|"
                      "CANCELLED(c@/!)")
                     )
 org-support-shift-select t
 org-modern-block-name nil
 org-speed-command-hook '(org-speed-command-activate
                          org-babel-speed-command-activate)
 org-modern-hide-stars nil
 org-modern-mode-hook '(org-modern-mode-set-explicitly)
 org-fold-core-isearch-open-function 'org-fold--isearch-reveal
 org-cite-export-processors '((beamer natbib) (latex biblatex) (csl csl))
 org-persist-before-write-hook '(org-element--cache-persist-before-write)
 org-tab-first-hook '(org-babel-hide-result-toggle-maybe
                      org-babel-header-arg-expand)
 org-link-shell-confirm-function 'yes-or-no-p
 org-babel-pre-tangle-hook '(org-decrypt-entries)
 org-agenda-loop-over-headlines-in-active-region nil
 org-todo-keyword-faces '(("NEXT" :inherit warning)
                          ("PROJECT" :inherit font-lock-string-face))
 org-cite-global-bibliography '("/home/huzf/plugable/share/bib/global.bib")
 org-fold-reveal-start-hook '(org-decrypt-entry)
 org-occur-hook '(org-first-headline-recenter)
 org-metadown-hook '(org-babel-pop-to-session-maybe)
 org-link-parameters '(("attachment" :follow org-attach-follow :complete
                        org-attach-complete-link)
                       ("id" :follow org-id-open)
                       ("eww" :follow org-eww-open :store org-eww-store-link)
                       ("rmail" :follow org-rmail-open :store
                        org-rmail-store-link)
                       ("mhe" :follow org-mhe-open :store org-mhe-store-link)
                       ("irc" :follow org-irc-visit :store org-irc-store-link
                        :export org-irc-export)
                       ("info" :follow org-info-open :export org-info-export
                        :store org-info-store-link :insert-description
                        org-info-description-as-command)
                       ("gnus" :follow org-gnus-open :store
                        org-gnus-store-link)
                       ("docview" :follow org-docview-open :export
                        org-docview-export :store org-docview-store-link)
                       ("bbdb" :follow org-bbdb-open :export org-bbdb-export
                        :complete org-bbdb-complete-link :store
                        org-bbdb-store-link)
                       ("w3m" :store org-w3m-store-link)
                       ("doi" :follow org-link-doi-open :export
                        org-link-doi-export)
                       ("bibtex" :follow org-bibtex-open :store
                        org-bibtex-store-link)
                       ("file+sys") ("file+emacs")
                       ("shell" :follow org-link--open-shell)
                       ("news" :follow
                        #[514 "\301\300\302\x04Q\x02\"\207"
                          ["news" browse-url ":"] 6 "\n\n(fn URL ARG)"]
                        )
                       ("mailto" :follow
                        #[514 "\301\300\302\x04Q\x02\"\207"
                          ["mailto" browse-url ":"] 6 "\n\n(fn URL ARG)"]
                        )
                       ("https" :follow
                        #[514 "\301\300\302\x04Q\x02\"\207"
                          ["https" browse-url ":"] 6 "\n\n(fn URL ARG)"]
                        )
                       ("http" :follow
                        #[514 "\301\300\302\x04Q\x02\"\207"
                          ["http" browse-url ":"] 6 "\n\n(fn URL ARG)"]
                        )
                       ("ftp" :follow
                        #[514 "\301\300\302\x04Q\x02\"\207" ["ftp" browse-url ":"]
                          6 "\n\n(fn URL ARG)"]
                        )
                       ("help" :follow org-link--open-help :store
                        org-link--store-help)
                       ("file" :complete org-link-complete-file)
                       ("elisp" :follow org-link--open-elisp))
 org-metaup-hook '(org-babel-load-in-session-maybe)
 org-modern-table nil
 org-refile-allow-creating-parent-nodes 'confirm
 org-todo-repeat-to-state "NEXT"
 org-outline-path-complete-in-steps nil
 org-fast-tag-selection-single-key 'expert
 org-tags-column -80
 org-refile-target-verify-function '(closure (t) nil
                                     (not
                                      (member (nth 2 (org-heading-components))
                                       org-done-keywords)
                                      )
                                     )
 )

Re: [BUG] Incorrect indentation when there are invisible/diplay properties on the line [9.6.7 ( @ /home/huzf/.cache/emacs_configs/default/elpa.28/org-9.6.7/)]

[Ihor Radchenko]

Hammer Hu <hammer@posteo.net> writes:

> When `org-modern' is installed and `org-modern-mode' is enable in a
> buffer, the fontification is augmented. See
> [[https://raw.githubusercontent.com/minad/org-modern/screenshots/example.gif]]
> for a demonstration.  But when I have `org-modern-mode' turned on,
> `org-adapt-indentation' to be `t', and indent some region, it becomes:
> ...
> #+begin_quote
> Thanks. I can reproduce the problem. It seems that `org-indent-region'
> or `org-indent-line' has issues computing the indentation if there are
> invisible/diplay properties on the line. This is not something we can
> fix in org-modern, since `org-modern' only augments the fontification
> and does not override indentation functionality. It should be repaired
> in Org directly if even - maybe ask on the Org issue tracker? I suggest
> you either don't use `org-adapt-identation' or set
> `org-modern-block-name' to nil. With `org-modern-block-name=nil' the
> issue also seems to go away.
> #+end_quote
>
> Emacs  : GNU Emacs 28.2.50 (build 1, x86_64-pc-linux-gnu, GTK+ Version 3.24.36, cairo version 1.17.6)
>  of 2023-01-07

Thanks for reporting!

Org relies on `current-indentation' to determine indentation.
AFAIU, `current-indentation' was not able to handle invisible and
display properties before Emacs 29.

May you please try to reproduce using Emacs 29 or newer?

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: [BUG] Incorrect indentation when there are invisible/diplay properties on the line [9.6.7 ( @ /home/huzf/.cache/emacs_configs/default/elpa.28/org-9.6.7/)]

[Ihor Radchenko]

[ Adding Org mailing list back to CC. Please use "reply all" ]

Hammer Hu <hammer@posteo.net> writes:

> I tried on the latest version, but the problem persists:
>
> Emacs  : GNU Emacs 30.0.50 (build 1, x86_64-pc-linux-gnu, GTK+ Version 3.24.38, cairo version 1.17.8)
>  of 2023-07-11
> Package: Org mode version 9.6.6 (release_9.6.6 @ /usr/share/emacs/30.0.50/lisp/org/)

Sorry, but I am unable to reproduce.

I tried the following:

1. Create /tmp/bug.org

** A level-2 heading
something

something

#+begin_quote
a block
#+end_quote

and something

2. Create bug.el

(add-hook 'org-mode-hook #'org-indent-mode)
(add-hook 'org-mode-hook #'org-modern-mode)


3. emacs -Q -L /path/to/compat/ -L /path/to/org-modern/ -l compat -l org-modern -l /tmp/bug.el /tmp/bug.org

4. Select region containing the code block
5. M-x indent-region <RET>
6. Nothing happens

I tried emacs master, emacs 28, and emacs 27.

Please, provide more detailed reproducer.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: [BUG] Incorrect indentation when there are invisible/diplay properties on the line [9.6.7 ( @ /home/huzf/.cache/emacs_configs/default/elpa.28/org-9.6.7/)]

[Hammer Hu]

Please change bug.el to

(add-hook 'org-mode-hook #'org-modern-mode)
(setq-default org-adapt-indentation t)

Select the region and M-x indent-region <RET> multiple times.

Best,

---------------------------------------------
From: Ihor Radchenko <yantar92@posteo.net>
Sent: 11.07.2023 13:22
To: Hammer Hu <hammer@posteo.net>
Subject: Re: [BUG] Incorrect indentation when there are invisible/diplay properties on the line [9.6.7 ( @ /home/huzf/.cache/emacs_configs/default/elpa.28/org-9.6.7/)]

[ Adding Org mailing list back to CC. Please use "reply all" ]

Hammer Hu <hammer@posteo.net> writes:

> I tried on the latest version, but the problem persists:
>
> Emacs  : GNU Emacs 30.0.50 (build 1, x86_64-pc-linux-gnu, GTK+ Version 3.24.38, cairo version 1.17.8)
>  of 2023-07-11
> Package: Org mode version 9.6.6 (release_9.6.6 @ /usr/share/emacs/30.0.50/lisp/org/)

Sorry, but I am unable to reproduce.

I tried the following:

1. Create /tmp/bug.org

** A level-2 heading
something

something

#+begin_quote
a block
#+end_quote

and something

2. Create bug.el

(add-hook 'org-mode-hook #'org-indent-mode)
(add-hook 'org-mode-hook #'org-modern-mode)


3. emacs -Q -L /path/to/compat/ -L /path/to/org-modern/ -l compat -l org-modern -l /tmp/bug.el /tmp/bug.org

4. Select region containing the code block
5. M-x indent-region <RET>
6. Nothing happens

I tried emacs master, emacs 28, and emacs 27.

Please, provide more detailed reproducer.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: [BUG] Incorrect indentation when there are invisible/diplay properties on the line [9.6.7 ( @ /home/huzf/.cache/emacs_configs/default/elpa.28/org-9.6.7/)]

[Ihor Radchenko]

Hammer Hu <hammer@posteo.net> writes:

> Please change bug.el to
>
> (add-hook 'org-mode-hook #'org-modern-mode)
> (setq-default org-adapt-indentation t)
>
> Select the region and M-x indent-region <RET> multiple times.

Thanks!

A simpler reproducer:

1. /tmp/bug.org

#+begin_quote
foo
#+end_quote

2. emacs -Q -L /path/to/compat/ -L /path/to/org-modern/ -l compat -l org-modern /tmp/bug.org

3. M-x org-modern-mode

4. Move to the beginning of #+begin_quote line

5. M-: (indent-line-to 3) <RET>
6. M-: (indent-line-to 3) <RET>
7. <can repeat>

8. M-x org-modern-mode
9. Observe overindentation

The reason why this happens is the following:

1. indent-line-to tries hard to create indentation and move the
   beginning of visible text to column 3.
2. indent-line-to notices (at point (5)) that line is not indented at
   all.
3. It computes that it should insert "   " to indent to column 3 and
   inserts these spaces.
4. org-modern-mode notices modification and re-hides spaces
5. indent-line-to is fires one more time at point (6)
6. indent-line-to notices that line is indented, but the leading
   whitespace is invisible.
7. It computes that it should yet insert extra "    " to move the text
   to column 3 visually (because the existing spaces are hidden).
8. It inserts the extra spaces
9. org-modern-mode notices modification and re-hides added spaces.

I believe that it is org-modern's fault. Indentation works are it
supposed to and tried hard to align text visually to third column.
org-modern fights against.

Note that indenting visually is Emacs' convention that applies
everywhere.

Canceled.
Not an Org mode bug.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: [BUG] Incorrect indentation when there are invisible/diplay properties on the line [9.6.7 ( @ /home/huzf/.cache/emacs_configs/default/elpa.28/org-9.6.7/)]

[Hammer Hu]

Thanks for your accommodations. Is it possible to introduce a feature to align
all blocks to the begining to the lines when indenting? I thing it helps when
copying contents from a org file without emacs installed.

Best,

---------------------------------------------
From: Ihor Radchenko <yantar92@posteo.net>
Sent: 11.07.2023 15:07
To: Hammer Hu <hammer@posteo.net>
Subject: Re: [BUG] Incorrect indentation when there are invisible/diplay properties on the line [9.6.7 ( @ /home/huzf/.cache/emacs_configs/default/elpa.28/org-9.6.7/)]

Hammer Hu <hammer@posteo.net> writes:

> Please change bug.el to
>
> (add-hook 'org-mode-hook #'org-modern-mode)
> (setq-default org-adapt-indentation t)
>
> Select the region and M-x indent-region <RET> multiple times.

Thanks!

A simpler reproducer:

1. /tmp/bug.org

#+begin_quote
foo
#+end_quote

2. emacs -Q -L /path/to/compat/ -L /path/to/org-modern/ -l compat -l org-modern /tmp/bug.org

3. M-x org-modern-mode

4. Move to the beginning of #+begin_quote line

5. M-: (indent-line-to 3) <RET>
6. M-: (indent-line-to 3) <RET>
7. <can repeat>

8. M-x org-modern-mode
9. Observe overindentation

The reason why this happens is the following:

1. indent-line-to tries hard to create indentation and move the
   beginning of visible text to column 3.
2. indent-line-to notices (at point (5)) that line is not indented at
   all.
3. It computes that it should insert "   " to indent to column 3 and
   inserts these spaces.
4. org-modern-mode notices modification and re-hides spaces
5. indent-line-to is fires one more time at point (6)
6. indent-line-to notices that line is indented, but the leading
   whitespace is invisible.
7. It computes that it should yet insert extra "    " to move the text
   to column 3 visually (because the existing spaces are hidden).
8. It inserts the extra spaces
9. org-modern-mode notices modification and re-hides added spaces.

I believe that it is org-modern's fault. Indentation works are it
supposed to and tried hard to align text visually to third column.
org-modern fights against.

Note that indenting visually is Emacs' convention that applies
everywhere.

Canceled.
Not an Org mode bug.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: [BUG] Incorrect indentation when there are invisible/diplay properties on the line [9.6.7 ( @ /home/huzf/.cache/emacs_configs/default/elpa.28/org-9.6.7/)]

[Ihor Radchenko]

Hammer Hu <hammer@posteo.net> writes:

> Thanks for your accommodations. Is it possible to introduce a feature to align
> all blocks to the begining to the lines when indenting? I thing it helps when
> copying contents from a org file without emacs installed.

May you elaborate? 

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: [BUG] Incorrect indentation when there are invisible/diplay properties on the line [9.6.7 ( @ /home/huzf/.cache/emacs_configs/default/elpa.28/org-9.6.7/)]

[Hammer Hu]

When indent-region, instead of getting:

---
** heading

   text 1

   #+begin_quote
   block
   #+end_quote

   text 2
---

we allow the it to be:

---
** heading

   text 1

#+begin_quote
block
#+end_quote

   text 2
---

It makes it easy to copy code snippets from org file without emacs. I put
together casual numerical codes with other research records in org files and
have them synced to cloud drive. So sometimes when I need to extract codes from
org files and don't have access to a computer with emacs, I either bear with
messy codes or adjust then manually. Even worse, when python codes is in need,
it would be a disaster.

Best,

---------------------------------------------
From: Ihor Radchenko <yantar92@posteo.net>
Sent: 12.07.2023 08:42
To: Hammer Hu <hammer@posteo.net>
Subject: Re: [BUG] Incorrect indentation when there are invisible/diplay properties on the line [9.6.7 ( @ /home/huzf/.cache/emacs_configs/default/elpa.28/org-9.6.7/)]

Hammer Hu <hammer@posteo.net> writes:

> Thanks for your accommodations. Is it possible to introduce a feature to align
> all blocks to the begining to the lines when indenting? I thing it helps when
> copying contents from a org file without emacs installed.

May you elaborate? 

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: [BUG] Incorrect indentation when there are invisible/diplay properties on the line [9.6.7 ( @ /home/huzf/.cache/emacs_configs/default/elpa.28/org-9.6.7/)]

[Ihor Radchenko]

Hammer Hu <hammer@posteo.net> writes:

> ** heading
>
>    text 1
>
> #+begin_quote
> block
> #+end_quote
>
>    text 2
> ---
>
> It makes it easy to copy code snippets from org file without emacs.

Check out org-src-preserve-indentation.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>