bug#45915: 27.1; deletechar distorts org-table

bug#45915: 27.1; deletechar distorts org-table

[Tak Kunihiro]

I am using emacs 27.1 with orgtbl-mode.  On certain condition, typing
<deletechar> distorts alighment of a table.  Then, typing <up> moves
previous line but far right position.

|   | rownames |     |   | rownames |
| / | <l>      |     | / | <l>      |
|---+----------| ->  |---+----------|
| / | a        |     |  | a        |

Here is a recipe to reproduce the glitch started from emcas -Q.  I
hope that together with bug#45091, this would be fixed before 27.2.

1. Create a buffer with (text-mode) and yank following table.

|   | rownames |
| / | <l>      |
|---+----------|
|   | a        |

2. (progn (require 'org) (call-interactively 'orgtbl-mode))
3. Move point to the first column and the third line with letter 'a'.
4. Hit <S-return>.
5. Hit <deletechar>.
6. Hit <up>.

bug#45915: 28.2; delete-char deletes two letters

[Ihor Radchenko]

Tak Kunihiro <homeros.misasa@gmail.com> writes:

> I confirmed that delete-char still deletes two letters on
> emacs-version: 28.2, org-version: 9.5.5.
>
> I found out (org-table-align) in (org-table-copy-down) makes
> this situation.
>
> Can someone take a look (org-table-align)?

I think that the thread has been lost because the bug has been assigned
to Org. I now tried to reassign the bug back to emacs.

-- 
Ihor Radchenko,
Org mode contributor,
Learn more about Org mode at https://orgmode.org/.
Support Org development at https://liberapay.com/org-mode,
or support my work at https://liberapay.com/yantar92

bug#45915: 28.2; delete-char deletes two letters

[Tak Kunihiro]

I see a light.
With the following patch, <delete> deletes only a letter.

The (org-table-align) inserts string with text properties.
It seems that the text properties matter.

However, I do not understand what's going on there. Can you take a look
(org-table-align) with this hint and revise it in correct way?

diff --git a/lisp/org/org-table.el b/lisp/org/org-table.el
index 9b692d09736..55bc2838718 100644
--- a/lisp/org/org-table.el
+++ b/lisp/org/org-table.el
@@ -4410,6 +4410,7 @@ org-table-align
 					             alignments)
 				          "|")))
 		           "|")))
+              (setq new (substring-no-properties new))
 	      (if (equal new previous)
 		  (forward-line)
 		(insert new "\n")

bug#45915: 28.2; delete-char deletes two letters

[Eli Zaretskii]

> Cc: tkk@misasa.okayama-u.ac.jp, Ihor Radchenko <yantar92@gmail.com>
> From: Tak Kunihiro <homeros.misasa@gmail.com>
> Date: Sat, 17 Sep 2022 12:48:56 +0900
> 
> I see a light.
> With the following patch, <delete> deletes only a letter.
> 
> The (org-table-align) inserts string with text properties.
> It seems that the text properties matter.
> 
> However, I do not understand what's going on there. Can you take a look
> (org-table-align) with this hint and revise it in correct way?

Why do you think your proposed change is incorrect?

Rather than asking us to study how org-table works and why it
misbehaves in this case, would it be possible for some Org expert to
explain that in simple terms, so that the relation of what happens to
the relevant core Emacs features could be easier to realize?  For
example, what are the text properties involved in this case, the ones
that substring-no-properties removes?  Is it possible that one of
these properties is 'composition'?

bug#45915: 28.2; delete-char deletes two letters

[Tak Kunihiro]

Eli Zaretskii <eliz@gnu.org> writes:

>> Cc: tkk@misasa.okayama-u.ac.jp, Ihor Radchenko <yantar92@gmail.com>
>> From: Tak Kunihiro <homeros.misasa@gmail.com>
>> Date: Sat, 17 Sep 2022 12:48:56 +0900
>> 
>> I see a light.
>> With the following patch, <delete> deletes only a letter.
>> 
>> The (org-table-align) inserts string with text properties.
>> It seems that the text properties matter.
>> 
>> However, I do not understand what's going on there. Can you take a look
>> (org-table-align) with this hint and revise it in correct way?
>
> Why do you think your proposed change is incorrect?
>
> Rather than asking us to study how org-table works and why it
> misbehaves in this case, would it be possible for some Org expert to
> explain that in simple terms, so that the relation of what happens to
> the relevant core Emacs features could be easier to realize?  For
> example, what are the text properties involved in this case, the ones
> that substring-no-properties removes?  Is it possible that one of
> these properties is 'composition'?

Dear Eli, thank you for the response. I really want to fix it to
switch to 28.2 from 26.3!

1. Following function will create a problematic org-table.

(defun emacs-bug-reproduce-45915 ()
  "Reproduce bug#45915."
  (interactive)
  (with-current-buffer (get-buffer-create "*temp buffer*")
    (erase-buffer)
    (require 'org)
    (orgtbl-mode 1)
    (setq org-table-copy-increment nil)
    (insert (format "emacs-version: %s, org-version: %s\n" emacs-version org-version))
    (insert "| 1 |"))
  (switch-to-buffer-other-window "*temp buffer*")
  (execute-kbd-macro (kbd "M-< C-n C-f C-f"))
  (message "I will call #'org-table-copy-down.")
  (call-interactively #'org-table-copy-down))
;;; (call-interactively 'emacs-bug-reproduce-45915)

2. Buffer is with the 1x2 table as shown below. <delete> deletes one
letter on the first row, and it does two letters on the
second row.

  emacs-version: 28.2, org-version: 9.5.5
  | 1 |
  | 1 |

3. I wonder what is the difference between two lines.
I examined each line with following statement.

  (buffer-substring (line-beginning-position) (line-end-position))

line 1 returns: "| 1 |"
line 2 returns: #("| 1 |" 1 2 (display (space :relative-width 1)) 3 4
(display (space :relative-width 1)))

4. Although I do not know how to understand the return for line 2,
I speculated the problem is something to do with text properties.

5. It is (org-table-align) that generates and inserts the second line
with text properties.

6. I tried to remove text-properties from string to be inserted by
(org-table-align) using substring-no-properties (as shown in the
patch). Now, both lines returns the same "| 1 |". Problem disappeared.

7. Since I do not know intention why (org-table-align) wants to give
text properties to tables, I cannot tell the side effect of the patch.

TIA

bug#45915: 28.2; delete-char deletes two letters

[Eli Zaretskii]

> From: Tak Kunihiro <homeros.misasa@gmail.com>
> Cc: Tak Kunihiro <homeros.misasa@gmail.com>,  45915@debbugs.gnu.org,
>   yantar92@gmail.com
> Cc: tkk@misasa.okayama-u.ac.jp
> Date: Sat, 17 Sep 2022 17:20:26 +0900
> 
> 1. Following function will create a problematic org-table.
> 
> (defun emacs-bug-reproduce-45915 ()
>   "Reproduce bug#45915."
>   (interactive)
>   (with-current-buffer (get-buffer-create "*temp buffer*")
>     (erase-buffer)
>     (require 'org)
>     (orgtbl-mode 1)
>     (setq org-table-copy-increment nil)
>     (insert (format "emacs-version: %s, org-version: %s\n" emacs-version org-version))
>     (insert "| 1 |"))
>   (switch-to-buffer-other-window "*temp buffer*")
>   (execute-kbd-macro (kbd "M-< C-n C-f C-f"))
>   (message "I will call #'org-table-copy-down.")
>   (call-interactively #'org-table-copy-down))
> ;;; (call-interactively 'emacs-bug-reproduce-45915)
> 
> 2. Buffer is with the 1x2 table as shown below. <delete> deletes one
> letter on the first row, and it does two letters on the
> second row.
> 
>   emacs-version: 28.2, org-version: 9.5.5
>   | 1 |
>   | 1 |

No, it doesn't delete 2 characters in the 2nd line, it deletes just
one.  After you press <Delete>, type C-b (to go to the space between
the two '|' characters, and type "C-x =".  You will see this:

  Char: SPC (32, #o40, #x20, part of display "  ") point=51 of 54 (93%) column=1

Note the "51 to 54" part: there's a 'display' property there that
covers _two_ buffer positions, not one.  That's because the two
'display' properties on the two SPC characters before the deletion
have the same value:

>   (buffer-substring (line-beginning-position) (line-end-position))
> 
> line 1 returns: "| 1 |"
> line 2 returns: #("| 1 |" 1 2 (display (space :relative-width 1)) 3 4
> (display (space :relative-width 1)))

Those two 'display' properties are now displayed as a single stretch
glyph, because two adjacent text properties with the same value are
indistinguishable from a single one that spans all of the buffer
positions.

This is not a bug, this is how Emacs always worked.

The question is now why is this a problem for you, and how it affects
Org Table.  The solution, if there's a need for one, should IMO come
from the Org side.

bug#45915: 28.2; delete-char deletes two letters

[Tak Kunihiro]

>>   (buffer-substring (line-beginning-position) (line-end-position))
>> 
>> line 1 returns: "| 1 |"
>> line 2 returns: #("| 1 |" 1 2 (display (space :relative-width 1)) 3 4
>> (display (space :relative-width 1)))
>
> Those two 'display' properties are now displayed as a single stretch
> glyph, because two adjacent text properties with the same value are
> indistinguishable from a single one that spans all of the buffer
> positions.
>
> This is not a bug, this is how Emacs always worked.

I see. I understand that 'display' properties on an org-table matters.
I took a look for (org-table-align) and found that
`org-table-separator-space' defines the text properties as shown below.

org-table.el: (defconst org-table-separator-space
org-table.el:   (propertize " " 'display '(space :relative-width 1))
org-table.el:   "Space used around fields when aligning the table.
org-table.el: This space serves as a segment separator for the purposes of the
org-table.el: bidirectional reordering.")

Although I still do not understand the purpose of `:relative-width',
problem disappeared with following configuration.

  (with-eval-after-load "org-table"
    (setq org-table-separator-space " "))

I have an impression that default characters on
org-table-separator-space is problematic.

bug#45915: 28.2; delete-char deletes two letters

[Eli Zaretskii]

> From: Tak Kunihiro <homeros.misasa@gmail.com>
> Cc: tkk@misasa.okayama-u.ac.jp
> Date: Sun, 18 Sep 2022 14:44:59 +0900
> 
> I see. I understand that 'display' properties on an org-table matters.
> I took a look for (org-table-align) and found that
> `org-table-separator-space' defines the text properties as shown below.
> 
> org-table.el: (defconst org-table-separator-space
> org-table.el:   (propertize " " 'display '(space :relative-width 1))
> org-table.el:   "Space used around fields when aligning the table.
> org-table.el: This space serves as a segment separator for the purposes of the
> org-table.el: bidirectional reordering.")

Yes, and that's exactly right for when the table could include
bidirectional (i.e. mixed L2R and R2L) text.

> Although I still do not understand the purpose of `:relative-width',
> problem disappeared with following configuration.
> 
>   (with-eval-after-load "org-table"
>     (setq org-table-separator-space " "))

I don't think this is the right solution.  One possible solution that
is better is to bind <Delete> to an org-table specific command, which
would insert some character displayed as a thin space between the two
spaces that have the 'display' property.  But I'll let Ihor chime in.

> I have an impression that default characters on
> org-table-separator-space is problematic.

What do you mean by "default characters"?

bug#45915: 28.2; delete-char deletes two letters

[Eli Zaretskii]

> Cc: tkk@misasa.okayama-u.ac.jp, yantar92@gmail.com, 45915@debbugs.gnu.org
> Date: Sun, 18 Sep 2022 08:56:48 +0300
> From: Eli Zaretskii <eliz@gnu.org>
> 
> I don't think this is the right solution.  One possible solution that
> is better is to bind <Delete> to an org-table specific command, which
> would insert some character displayed as a thin space between the two
> spaces that have the 'display' property.  But I'll let Ihor chime in.

One other possibility is to use a slightly different :relative-width
factor for the two spaces in a table cell: one with the value of 1,
the other with 1.001 (say).  They will be indistinguishable on
display, but since the values are not equal, both stretch gfyphs will
be displayed, not just one.

bug#45915: 28.2; delete-char deletes two letters

[Tak Kunihiro]

> One other possibility is to use a slightly different :relative-width
> factor for the two spaces in a table cell: one with the value of 1,
> the other with 1.001 (say).  They will be indistinguishable on
> display, but since the values are not equal, both stretch gfyphs will
> be displayed, not just one.

Function (org-table--align-field) in org-table.el sandwiches 
text by org-table-separator-space as shown below.

    (concat org-table-separator-space
 	    prefix
 	    field
 	    suffix
-	    org-table-separator-space)))
+	    (propertize org-table-separator-space 'display '(space :relative-width 1.001)))))

I confirmed that this modification works good.

>> I have an impression that default characters on
>> org-table-separator-space is problematic.
>
> What do you mean by "default characters"?

Let me take the statement back.

bug#45915: 28.2; delete-char deletes two letters

[Ihor Radchenko]

Eli Zaretskii <eliz@gnu.org> writes:

>> I don't think this is the right solution.  One possible solution that
>> is better is to bind <Delete> to an org-table specific command, which
>> would insert some character displayed as a thin space between the two
>> spaces that have the 'display' property.  But I'll let Ihor chime in.
>
> One other possibility is to use a slightly different :relative-width
> factor for the two spaces in a table cell: one with the value of 1,
> the other with 1.001 (say).  They will be indistinguishable on
> display, but since the values are not equal, both stretch gfyphs will
> be displayed, not just one.

This is a good idea. I used it to fix the reported issue.

Fixed on main.
https://git.savannah.gnu.org/cgit/emacs/org-mode.git/commit/?id=9dde82411aefe5a6728eef95c3629dbf266fefa5

-- 
Ihor Radchenko,
Org mode contributor,
Learn more about Org mode at https://orgmode.org/.
Support Org development at https://liberapay.com/org-mode,
or support my work at https://liberapay.com/yantar92

bug#45915: 28.2; delete-char deletes two letters

[Eli Zaretskii]

> From: Ihor Radchenko <yantar92@gmail.com>
> Cc: homeros.misasa@gmail.com,  tkk@misasa.okayama-u.ac.jp,
>   45915@debbugs.gnu.org, emacs-orgmode@gnu.org
> Date: Thu, 22 Sep 2022 20:03:55 +0800
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > One other possibility is to use a slightly different :relative-width
> > factor for the two spaces in a table cell: one with the value of 1,
> > the other with 1.001 (say).  They will be indistinguishable on
> > display, but since the values are not equal, both stretch gfyphs will
> > be displayed, not just one.
> 
> This is a good idea. I used it to fix the reported issue.
> 
> Fixed on main.

Great.  So I guess this bug report can be closed?

bug#45915: 28.2; delete-char deletes two letters

[Tak Kunihiro]

>>> One other possibility is to use a slightly different :relative-width
>>> factor for the two spaces in a table cell: one with the value of 1,
>>> the other with 1.001 (say).  They will be indistinguishable on
>>> display, but since the values are not equal, both stretch gfyphs will
>>> be displayed, not just one.
>> 
>> This is a good idea. I used it to fix the reported issue.
>> 
>> Fixed on main.
> 
> Great.  So I guess this bug report can be closed?

Nice. Yes, please close this bug report.

bug#45915: 28.2; delete-char deletes two letters

[Eli Zaretskii]

> Date: Fri, 23 Sep 2022 06:36:58 +0900 (JST)
> Cc: yantar92@gmail.com, homeros.misasa@gmail.com, 45915@debbugs.gnu.org,
>  emacs-orgmode@gnu.org, tkk@misasa.okayama-u.ac.jp
> From: Tak Kunihiro <tkk@misasa.okayama-u.ac.jp>
> 
> >>> One other possibility is to use a slightly different :relative-width
> >>> factor for the two spaces in a table cell: one with the value of 1,
> >>> the other with 1.001 (say).  They will be indistinguishable on
> >>> display, but since the values are not equal, both stretch gfyphs will
> >>> be displayed, not just one.
> >> 
> >> This is a good idea. I used it to fix the reported issue.
> >> 
> >> Fixed on main.
> > 
> > Great.  So I guess this bug report can be closed?
> 
> Nice. Yes, please close this bug report.

Thanks, done.

bug#45915: 28.2; delete-char deletes two letters

[Tak Kunihiro]

OK. Now I understand <delete> only deletes one letter.  Two spaces are
shown as one letter, as expected.

To have a distorted table soon after deletion of text is OK.

However, when two spaces are shown as one letter, point moves randomly
when I press <up>.  I cannot keep editing with this condition.

I show recipe to reproduce this situation. Is there a trick to move
point in expected way?

(defun emacs-bug-reproduce-45915 ()
  "Reproduce bug#45915."
  ;; (gnus-read-ephemeral-emacs-bug-group 45915)
  (interactive)
  (with-current-buffer (get-buffer-create "*temp buffer*")
    (erase-buffer)
    (require 'org)
    (orgtbl-mode 1)
    (insert (format "emacs-version: %s, org-version: %s\n" emacs-version org-version))
    (insert "| 1 |   |   |   |   |   |   |   |   |"))
  (switch-to-buffer-other-window "*temp buffer*")
  (execute-kbd-macro (kbd "M-< C-n C-f C-f"))
  (message "I will call #'org-table-copy-down.")
  (sit-for 2)
  (call-interactively #'org-table-copy-down)
  (sit-for 2)
  (message "I will hit <delete>.")
  (sit-for 2)
  (execute-kbd-macro (kbd "<delete>"))
  (message "One letter was deleted expectedly.")
  (sit-for 2)
  (message "I will hit <up>.")
  (sit-for 2)
  (execute-kbd-macro (kbd "<up>"))
  (message "Point moved unexpectedly."))
;;; (call-interactively 'emacs-bug-reproduce-45915)

bug#45915: 28.2; delete-char deletes two letters

[Eli Zaretskii]

> From: Tak Kunihiro <homeros.misasa@gmail.com>
> Cc: Tak Kunihiro <homeros.misasa@gmail.com>,  yantar92@gmail.com,
>   45915@debbugs.gnu.org
> Cc: tkk@misasa.okayama-u.ac.jp
> Date: Mon, 19 Sep 2022 10:24:55 +0900
> 
> However, when two spaces are shown as one letter, point moves randomly
> when I press <up>.  I cannot keep editing with this condition.
> 
> I show recipe to reproduce this situation. Is there a trick to move
> point in expected way?

This is an unrelated bug, now fixed on the master branch.

(A workaround is to press C-f or <RIGHT> before <UP>.)