bug#43587: 27.1; Org is breaking links

bug#43587: 27.1; Org is breaking links

[Christoph Arenz]

[-- Attachment #1: Type: text/plain, Size: 1352 bytes --]

Org-mode is sometimes breaking links while adjusting indentation in
org-fixup-indentation.
This has been observed on emacs version 27.1, with org-mode versions 9.3
(built-in) and 9.4 (melpa-stable).
It also occurs with the current emacs development version.

Here is how to recreate. I hope my mailer does not screw up the lines
below...
Enter the following text in an org buffer and proceed as described:

* Org is breaking links!
   Create a heading with `*' and random heading text.
   Create a text section below the heading.
   In the text section create a separate line starting with a link, like so:
   [[simple-link]]
   [[http://a.b.com/demo.html][link text]]
   Switch on whitespace-mode to see what is going on.
   Indent the heading deeper (i.e. demote) with several `M-<right>'
until the text starts with a TAB instead of SPCs.
   Then outdent the heading back (i.e. promote) with `M-<left>'.
   See the links getting corrupted as the two leftmost brackets `[[' are
removed, in case of links with a target part, this also gets removed.
   Note that his behavior also affects org-refile: Links can get can get
corrupted in this way while refiling.
   Note that links do not get broken as described when showing full
links with org-toggle-link-display.

Thanks for your help,
Kind Regards,
Christoph

[-- Attachment #2: Type: text/html, Size: 1899 bytes --]

bug#43587: move-to-column behaves differently when text has invisible property

[Christoph Arenz]

[-- Attachment #1: Type: text/plain, Size: 1014 bytes --]

Digging deeper into bug 43587, it seems to be related to move-to-column
behaving slightly different when the text in a line has the invisible
property set.

Here is how to recreate. According to the documentation for
move-to-column, I would not expect any difference between the two lines:

 > Optional second argument FORCE non-nil means if COLUMN is in the
middle of a tab character, change it to spaces.

For the first line, the tab has not been changed to spaces...

Am I overlooking something?

Kind Regards,
Christoph

(progn
   (switch-to-buffer "indent-test.txt")
   (erase-buffer)
   (insert "\tLine starting with INVISIBLE text after TAB\n")
   (insert "\tLine starting with visible text after TAB\n")
   (insert "\nUsing move-to-column to move 'into' TAB, using the FORCE
parameter on both lines\n")
   (whitespace-mode 1)
   (add-text-properties 2 21 '(invisible t))
   (beginning-of-buffer)
   (move-to-column 7 t)
   (forward-line)
   (move-to-column 7 t))


[-- Attachment #2: Type: text/html, Size: 1452 bytes --]

bug#43587: move-to-column behaves differently when text has invisible property

[Eli Zaretskii]

> From: Christoph Arenz <tiga.arenz@web.de>
> Date: Tue, 29 Sep 2020 17:56:45 +0200
> 
> Digging deeper into bug 43587, it seems to be related to move-to-column behaving slightly different when the
> text in a line has the invisible property set.
> 
> Here is how to recreate. According to the documentation for move-to-column, I would not expect any
> difference between the two lines:
> 
> > Optional second argument FORCE non-nil means if COLUMN is in the middle of a tab character, change
> it to spaces.
> 
> For the first line, the tab has not been changed to spaces...
> 
> Am I overlooking something?

Yes, move-to-column skips invisible text.

bug#43587: move-to-column behaves differently when text has invisible property

[Eli Zaretskii]

[Please use Reply All to CC the bug address as well.]

> From: Christoph Arenz <tiga.arenz@web.de>
> Date: Tue, 29 Sep 2020 18:29:22 +0200
> 
> 
> On 29.09.20 18:22, Eli Zaretskii wrote:
> >> From: Christoph Arenz <tiga.arenz@web.de>
> >> Date: Tue, 29 Sep 2020 17:56:45 +0200
> >>
> >> Digging deeper into bug 43587, it seems to be related to move-to-column behaving slightly different when the
> >> text in a line has the invisible property set.
> >>
> >> Here is how to recreate. According to the documentation for move-to-column, I would not expect any
> >> difference between the two lines:
> >>
> >>> Optional second argument FORCE non-nil means if COLUMN is in the middle of a tab character, change
> >> it to spaces.
> >>
> >> For the first line, the tab has not been changed to spaces...
> >>
> >> Am I overlooking something?
> > Yes, move-to-column skips invisible text.
> Given that there is no invisible text in columns 0 to 7 in the provided
> example, but only thereafter(!), I do not understand what 'skips' means
> in this context when moving to column 7...

Sorry, I've misunderstood what you were asking.  I thought you were
asking whether move-to-column is affected by invisible text, and
answered that question.

bug#43587: move-to-column behaves differently when text has invisible property

[Eli Zaretskii]

> From: Christoph Arenz <tiga.arenz@web.de>
> Date: Tue, 29 Sep 2020 17:56:45 +0200
> 
> (progn
>   (switch-to-buffer "indent-test.txt")
>   (erase-buffer)
>   (insert "\tLine starting with INVISIBLE text after TAB\n")
>   (insert "\tLine starting with visible text after TAB\n")
>   (insert "\nUsing move-to-column to move 'into' TAB, using the FORCE parameter on both lines\n")
>   (whitespace-mode 1)
>   (add-text-properties 2 21 '(invisible t))
>   (beginning-of-buffer)
>   (move-to-column 7 t)
>   (forward-line)
>   (move-to-column 7 t))

It's a deficiency in the algorithm used by move-to-column that
determines whether the goal column is in the middle of a TAB.  It
assumes that in such a case the TAB is the previous character that
ends at the column where the movement ended, but that is not so when
invisible text immediately follows the TAB, because moving over the
TAB will in that case also skip all of the following invisible text.

I will see how this can be fixed without making the function too slow
(because it needs to consider overlays as well, not just text
properties).

bug#43587: move-to-column behaves differently when text has invisible property

[Eli Zaretskii]

> Date: Tue, 29 Sep 2020 22:00:24 +0300
> From: Eli Zaretskii <eliz@gnu.org>
> Cc: 43587@debbugs.gnu.org
> 
> It's a deficiency in the algorithm used by move-to-column that
> determines whether the goal column is in the middle of a TAB.  It
> assumes that in such a case the TAB is the previous character that
> ends at the column where the movement ended, but that is not so when
> invisible text immediately follows the TAB, because moving over the
> TAB will in that case also skip all of the following invisible text.
> 
> I will see how this can be fixed without making the function too slow
> (because it needs to consider overlays as well, not just text
> properties).

I think I fixed this now on the master branch.

bug#43587: move-to-column behaves differently when text has invisible property

[Christoph Arenz]

On 30.09.20 16:35, Eli Zaretskii wrote:

> I think I fixed this now on the master branch.

For the initial problem report, everything looks fine to me now. This
can be closed.

Thank you Eli!

Kind Regards, Christoph

bug#43587: move-to-column behaves differently when text has invisible property

[Eli Zaretskii]

> Cc: 43587@debbugs.gnu.org
> From: Christoph Arenz <tiga.arenz@web.de>
> Date: Tue, 6 Oct 2020 18:01:54 +0200
> 
> On 30.09.20 16:35, Eli Zaretskii wrote:
> 
> > I think I fixed this now on the master branch.
> 
> For the initial problem report, everything looks fine to me now. This
> can be closed.
> 
> Thank you Eli!

Thanks, closing.