bug#45915: 27.1; deletechar distorts org-table

bug#45915: 27.1; deletechar distorts org-table

[Tak Kunihiro]

I am using emacs 27.1 with orgtbl-mode.  On certain condition, typing
<deletechar> distorts alighment of a table.  Then, typing <up> moves
previous line but far right position.

|   | rownames |     |   | rownames |
| / | <l>      |     | / | <l>      |
|---+----------| ->  |---+----------|
| / | a        |     |  | a        |

Here is a recipe to reproduce the glitch started from emcas -Q.  I
hope that together with bug#45091, this would be fixed before 27.2.

1. Create a buffer with (text-mode) and yank following table.

|   | rownames |
| / | <l>      |
|---+----------|
|   | a        |

2. (progn (require 'org) (call-interactively 'orgtbl-mode))
3. Move point to the first column and the third line with letter 'a'.
4. Hit <S-return>.
5. Hit <deletechar>.
6. Hit <up>.

bug#45915: 27.1.91; deletechar distorts org-table

[Tak Kunihiro]

I confirm that there still is following problem on 27.1.91.
Could someone take a look and show me work around?

> I am using emacs 27.1 with orgtbl-mode.  On certain condition, typing
> <deletechar> distorts alighment of a table.  Then, typing <up> moves
> previous line but far right position.
>
> |   | rownames |     |   | rownames |
> | / | <l>      |     | / | <l>      |
> |---+----------| ->  |---+----------|
> | / | a        |     |  | a        |
>
> Here is a recipe to reproduce the glitch started from emcas -Q.  I
> hope that together with bug#45091, this would be fixed before 27.2.
>
> 1. Create a buffer with (text-mode) and yank following table.
>
> |   | rownames |
> | / | <l>      |
> |---+----------|
> |   | a        |
>
> 2. (progn (require 'org) (call-interactively 'orgtbl-mode))
> 3. Move point to the first column and the third line with letter 'a'.
> 4. Hit <S-return>.
> 5. Hit <deletechar>.
> 6. Hit <up>.

bug#45915: 27.1.91; deletechar distorts org-table

[Bastien]

Tak Kunihiro <homeros.misasa@gmail.com> writes:

> I confirm that there still is following problem on 27.1.91.
> Could someone take a look and show me work around?

I cannot reproduce this with latest Org stable version 9.4.5.

Can you try again and report?

See https://orgmode.org/org.html#Installation on how to install
using Org’s git repository if you want to try the latest version.

bug#45915: 27.2; deletechar distorts org-table

[Tak Kunihiro]

>> I confirm that there still is following problem on 27.1.91.
>> Could someone take a look and show me work around?
> 
> I cannot reproduce this with latest Org stable version 9.4.5.
> 
> Can you try again and report?

I downloaded Org 9.4.5 and reproduced the problem on 26.3, 27.2,
and 28.0.50. Here is a recipe again.

> On certain condition, typing <deletechar> distorts alignment of
> a table.  Then, typing <up> moves point to previous line but far
> right position (column 14 instead of column 2).
> 
> |   | rownames |     |   | rownames |
> | / | <l>      |     | / | <l>      |
> |---+----------| ->  |---+----------|
> | / | a        |     |  | a        |
> 
> Here is a recipe to reproduce the glitch started from emcas -Q. 
> 
> 1. Create a buffer with (text-mode) and yank following table.
> 
> |   | rownames |
> | / | <l>      |
> |---+----------|
> |   | a        |
> 
> 2. (progn (require 'org) (call-interactively 'orgtbl-mode))
> 3. Move point to the first column and the third line with letter 'a'.
> 4. Hit <S-return>.
> 5. Hit <deletechar>.
> 6. Hit <up>.

bug#45915: 29.0.50; deletechar distorts org-table

[Tak Kunihiro]

I confirm that there is still problem with org-table
on Emacs 29.0.50 with org-version 9.5.
I created a function to produce the problem as shown below.
Can you try again?

(defun emacs-bug-reproduce-45915 ()
  "Reproduce bug#45915."
  ;; (gnus-read-ephemeral-emacs-bug-group 45915)
  (interactive)
  (with-current-buffer (get-buffer-create "*temp buffer*")
    (erase-buffer)
    (require 'org)
    (orgtbl-mode 1)
    (insert (format "emacs-version: %s, org-version: %s\n" emacs-version org-version))
    (insert "|   | rownames |
| / | <l>      |
|---+----------|
|   | a        |"))
  (switch-to-buffer-other-window "*temp buffer*")
  (execute-kbd-macro (kbd "M-< C-n C-n C-n C-n C-f C-f"))
  (message "I will hit S-<return>.")
  (sit-for 2)
  (execute-kbd-macro (kbd "S-<return>"))
  (message "I will hit <delete>.")
  (sit-for 2)
  (execute-kbd-macro (kbd "<delete>"))
  (message "I will call previous-line.")
  (sit-for 2)
  (call-interactively 'previous-line))
;;; (call-interactively 'emacs-bug-reproduce-45915)


>>> I confirm that there still is following problem on 27.1.91.
>>> Could someone take a look and show me work around?
>> 
>> I cannot reproduce this with latest Org stable version 9.4.5.
>> 
>> Can you try again and report?
>
> I downloaded Org 9.4.5 and reproduced the problem on 26.3, 27.2,
> and 28.0.50. Here is a recipe again.
>
>> On certain condition, typing <deletechar> distorts alignment of
>> a table.  Then, typing <up> moves point to previous line but far
>> right position (column 14 instead of column 2).
>> 
>> |   | rownames |     |   | rownames |
>> | / | <l>      |     | / | <l>      |
>> |---+----------| ->  |---+----------|
>> | / | a        |     |  | a        |
>> 
>> Here is a recipe to reproduce the glitch started from emcas -Q. 
>> 
>> 1. Create a buffer with (text-mode) and yank following table.
>> 
>> |   | rownames |
>> | / | <l>      |
>> |---+----------|
>> |   | a        |
>> 
>> 2. (progn (require 'org) (call-interactively 'orgtbl-mode))
>> 3. Move point to the first column and the third line with letter 'a'.
>> 4. Hit <S-return>.
>> 5. Hit <deletechar>.
>> 6. Hit <up>.

bug#45915: 28.0.90; deletechar distorts org-table

[Tak Kunihiro]

I confirm that there still is the problem with org-table on Emacs
28.0.90 pretest with org-version 9.5.1.

Here is a recipe to reproduce the problem.  I still have to stick to
26.3.

 1. emacs -Q
 2. (defun emacs-bug-reproduce-45915 () ...)
 3. (call-interactively 'emacs-bug-reproduce-45915)

(defun emacs-bug-reproduce-45915 ()
  "Reproduce bug#45915."
  ;; (gnus-read-ephemeral-emacs-bug-group 45915)
  (interactive)
  (with-current-buffer (get-buffer-create "*temp buffer*")
    (erase-buffer)
    (require 'org)
    (orgtbl-mode 1)
    (insert (format "emacs-version: %s, org-version: %s\n" emacs-version org-version))
    (insert "|   | rownames |
| / | <l>      |
|---+----------|
|   | a        |"))
  (switch-to-buffer-other-window "*temp buffer*")
  (execute-kbd-macro (kbd "M-< C-n C-n C-n C-n C-f C-f"))
  (message "I will hit S-<return>.")
  (sit-for 2)
  (execute-kbd-macro (kbd "S-<return>"))
  (message "I will hit <delete>.")
  (sit-for 2)
  (execute-kbd-macro (kbd "<delete>"))
  (message "I will call previous-line.")
  (sit-for 2)
  (call-interactively 'previous-line))

> I confirm that there is still problem with org-table
> on Emacs 29.0.50 with org-version 9.5.
>
>>>> I confirm that there still is following problem on 27.1.91.
>>>> Could someone take a look and show me work around?
>>> 
>>> I cannot reproduce this with latest Org stable version 9.4.5.
>>> 
>>> Can you try again and report?
>>
>> I downloaded Org 9.4.5 and reproduced the problem on 26.3, 27.2,
>> and 28.0.50. Here is a recipe again.
>>
>>> On certain condition, typing <deletechar> distorts alignment of
>>> a table.  Then, typing <up> moves point to previous line but far
>>> right position (column 14 instead of column 2).
>>> 
>>> |   | rownames |     |   | rownames |
>>> | / | <l>      |     | / | <l>      |
>>> |---+----------| ->  |---+----------|
>>> | / | a        |     |  | a        |
>>> 
>>> Here is a recipe to reproduce the glitch started from emcas -Q. 
>>> 
>>> 1. Create a buffer with (text-mode) and yank following table.
>>> 
>>> |   | rownames |
>>> | / | <l>      |
>>> |---+----------|
>>> |   | a        |
>>> 
>>> 2. (progn (require 'org) (call-interactively 'orgtbl-mode))
>>> 3. Move point to the first column and the third line with letter 'a'.
>>> 4. Hit <S-return>.
>>> 5. Hit <deletechar>.
>>> 6. Hit <up>.

bug#45915: 28.0.90; deletechar distorts org-table

[Ihor Radchenko]

[-- Attachment #1: Type: text/plain, Size: 309 bytes --]

Tak Kunihiro <homeros.misasa@gmail.com> writes:

> I confirm that there still is the problem with org-table on Emacs
> 28.0.90 pretest with org-version 9.5.1.
>
> Here is a recipe to reproduce the problem.  I still have to stick to
> 26.3.

I was able to reproduce. Can you try the attached fix?

Best,
Ihor


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-org-table.el-Fix-orgtbl-mode-not-remapping-delete-ke.patch --]
[-- Type: text/x-diff, Size: 1090 bytes --]

From 7b1a7d43e090292b6bd821c65a644f8c599b2044 Mon Sep 17 00:00:00 2001
Message-Id: <7b1a7d43e090292b6bd821c65a644f8c599b2044.1639918362.git.yantar92@gmail.com>
From: Ihor Radchenko <yantar92@gmail.com>
Date: Sun, 19 Dec 2021 20:49:54 +0800
Subject: [PATCH] org-table.el: Fix orgtbl-mode not remapping delete key

* lisp/org-table.el (orgtbl-setup): Remap `delete-forward-char' in
addition to `deleta-char'.  The former is called in Emacs >26.

Fixes https://orgmode.org/list/1AD1B9B7-5A15-4F86-9274-B04B83694C85@misasa.okayama-u.ac.jp
---
 lisp/org-table.el | 1 +
 1 file changed, 1 insertion(+)

diff --git a/lisp/org-table.el b/lisp/org-table.el
index 1546cd563..0560e32a1 100644
--- a/lisp/org-table.el
+++ b/lisp/org-table.el
@@ -5290,6 +5290,7 @@ (defun orgtbl-setup ()
       (org-remap orgtbl-mode-map
 		 'self-insert-command 'orgtbl-self-insert-command
 		 'delete-char 'org-delete-char
+                 'delete-forward-char 'org-delete-char
 		 'delete-backward-char 'org-delete-backward-char)
       (org-defkey orgtbl-mode-map "|" 'org-force-self-insert))
     t))
-- 
2.32.0

bug#45915: 28.0.90; deletechar distorts org-table

[Tak Kunihiro]

Dear Ihor

Thank you for the patch.  I applied the patch to org-table.el on Emacs
28.0.90.

And I still see the problem there.  Is the problem solved by the patch
on your environment?

Tak

Ihor Radchenko <yantar92@gmail.com> writes:

> Tak Kunihiro <homeros.misasa@gmail.com> writes:
>
>> I confirm that there still is the problem with org-table on Emacs
>> 28.0.90 pretest with org-version 9.5.1.
>>
>> Here is a recipe to reproduce the problem.  I still have to stick to
>> 26.3.
>
> I was able to reproduce. Can you try the attached fix?
>
> Best,
> Ihor
>
> From 7b1a7d43e090292b6bd821c65a644f8c599b2044 Mon Sep 17 00:00:00 2001
> Message-Id: <7b1a7d43e090292b6bd821c65a644f8c599b2044.1639918362.git.yantar92@gmail.com>
> From: Ihor Radchenko <yantar92@gmail.com>
> Date: Sun, 19 Dec 2021 20:49:54 +0800
> Subject: [PATCH] org-table.el: Fix orgtbl-mode not remapping delete key
>
> * lisp/org-table.el (orgtbl-setup): Remap `delete-forward-char' in
> addition to `deleta-char'.  The former is called in Emacs >26.
>
> Fixes https://orgmode.org/list/1AD1B9B7-5A15-4F86-9274-B04B83694C85@misasa.okayama-u.ac.jp
> ---
>  lisp/org-table.el | 1 +
>  1 file changed, 1 insertion(+)
>
> diff --git a/lisp/org-table.el b/lisp/org-table.el
> index 1546cd563..0560e32a1 100644
> --- a/lisp/org-table.el
> +++ b/lisp/org-table.el
> @@ -5290,6 +5290,7 @@ (defun orgtbl-setup ()
>        (org-remap orgtbl-mode-map
>  		 'self-insert-command 'orgtbl-self-insert-command
>  		 'delete-char 'org-delete-char
> +                 'delete-forward-char 'org-delete-char
>  		 'delete-backward-char 'org-delete-backward-char)
>        (org-defkey orgtbl-mode-map "|" 'org-force-self-insert))
>      t))

bug#45915: 28.0.90; deletechar distorts org-table

[Ihor Radchenko]

Tak Kunihiro <homeros.misasa@gmail.com> writes:

> Thank you for the patch.  I applied the patch to org-table.el on Emacs
> 28.0.90.
>
> And I still see the problem there.  Is the problem solved by the patch
> on your environment?

The problem was solved on my side, but _not_ using your reproducer
explicitly. If I save the buffer from your reproducer as a file and
later open in, the patch does solve the problem. However, when I run the
reproducer as is, a similar-looking problem emerges.

Using your reproducer, something very strange is going on.
With the patch, <delete> is correctly mapped to org-delete-char, which
calls (delete-char 1) at some point. However, this (delete-char 1)
somehow deletes 2!! chars instead of 1. This should be some tricky Emacs
issue (bug?). In any case, I am not ready to dig into what is going on
in Emacs. I do not think that such problem is going to appear during
normal Emacs usage. At least, I was unable to reproduce it by other
means than your reproducer function.

To illustrate the issue, one can use the following version of the
reproducer. Feel free to report this to Emacs upstream.

(defun emacs-bug-reproduce-45915 ()
  "Reproduce bug#45915."
  ;; (gnus-read-ephemeral-emacs-bug-group 45915)
  (interactive)
  (with-current-buffer (get-buffer-create "*temp buffer*")
    (erase-buffer)
    (require 'org)
    (orgtbl-mode 1)
    (insert (format "emacs-version: %s, org-version: %s\n" emacs-version org-version))
    (insert "|   | rownames |
| / | <l>      |
|---+----------|
|   | a        |"))
  (switch-to-buffer-other-window "*temp buffer*")
  (execute-kbd-macro (kbd "M-< C-n C-n C-n C-n C-f C-f"))
  (message "I will hit S-<return>.")
  (sit-for 2)
  (execute-kbd-macro (kbd "S-<return>"))
  (message "I will hit <delete>.")
  (sit-for 2)
  ;;  (execute-kbd-macro (kbd "<delete>"))
  (delete-char 1)
  (message "I will call previous-line.")
  (sit-for 2)
  (call-interactively 'previous-line))

Best,
Ihor

bug#45915: 28.0.90; deletechar distorts org-table

[Tak Kunihiro]

Dear Ihor,

> However, this (delete-char 1) somehow deletes 2!! chars instead of
> 1. This should be some tricky Emacs issue (bug?). In any case, I am
> not ready to dig into what is going on in Emacs. I do not think that
> such problem is going to appear during normal Emacs usage. At least, I
> was unable to reproduce it by other means than your reproducer
> function.
>
> To illustrate the issue, one can use the following version of the
> reproducer. Feel free to report this to Emacs upstream.

(defun emacs-bug-reproduce-45915 ()
  "Reproduce bug#45915."
  ;; (gnus-read-ephemeral-emacs-bug-group 45915)
  (interactive)
  (with-current-buffer (get-buffer-create "*temp buffer*")
    (erase-buffer)
    (require 'org)
    (orgtbl-mode 1)
    (insert (format "emacs-version: %s, org-version: %s\n" emacs-version org-version))
    (insert "|   | rownames |
| / | <l>      |
|---+----------|
|   | a        |"))
  (switch-to-buffer-other-window "*temp buffer*")
  (execute-kbd-macro (kbd "M-< C-n C-n C-n C-n C-f C-f"))
  (message "I will hit S-<return>.")
  (sit-for 2)
  (execute-kbd-macro (kbd "S-<return>"))
  (message "I will hit <delete>.")
  (sit-for 2)
  ;;  (execute-kbd-macro (kbd "<delete>"))
  (delete-char 1)
  (message "I will call previous-line.")
  (sit-for 2)
  (call-interactively 'previous-line))

I still see the problem with GNU Emacs 29.0.50 (build 8,
x86_64-apple-darwin21.5.0, NS appkit-2113.50 Version 12.4 (Build 21F79))
of 2022-05-23 with org-version 9.5.3.

I create org-tables and revise these everyday.  On duplication of a
cell, I hit S-<return> often. The problem occurs, with my use case, once
in 5 minutes. IOW point moves unexpectedly once in 5 minutes.

I think that there is an issue in `org-table-copy-down'. However, I
cannot tell what specifically is not good.

Is this issue of Emacs instead of Org?

bug#45915: 28.0.90; deletechar distorts org-table

[Ihor Radchenko]

Tak Kunihiro <homeros.misasa@gmail.com> writes:

> I still see the problem with GNU Emacs 29.0.50 (build 8,
> x86_64-apple-darwin21.5.0, NS appkit-2113.50 Version 12.4 (Build 21F79))
> of 2022-05-23 with org-version 9.5.3.
>
> I create org-tables and revise these everyday.  On duplication of a
> cell, I hit S-<return> often. The problem occurs, with my use case, once
> in 5 minutes. IOW point moves unexpectedly once in 5 minutes.

To clarify, I haven't applied the patch onto Org upstream.
Before the patch, the table is distorted because of an omission in Org.
After the patch, the table is distorted only using your reproducer
specifically, and the issue appears to be on Emacs side.

Best,
Ihor

bug#45915: 28.0.90; deletechar distorts org-table

[Ihor Radchenko]

Ihor Radchenko <yantar92@gmail.com> writes:

> To clarify, I haven't applied the patch onto Org upstream.
> Before the patch, the table is distorted because of an omission in Org.
> After the patch, the table is distorted only using your reproducer
> specifically, and the issue appears to be on Emacs side.

I now applied the patch onto main via 65e19a081.

This should solve part of the problem.
The remaining part should be solved on Emacs side (AFAIU).

Best,
Ihor

bug#45915: 28.0.90; deletechar distorts org-table

[Ihor Radchenko]

Tak Kunihiro <homeros.misasa@gmail.com> writes:

> Thank you for the patch.  I applied the patch to org-table.el on Emacs
> 28.0.90.
>
> And I still see the problem there.  Is the problem solved by the patch
> on your environment?

The problem was solved on my side, but _not_ using your reproducer
explicitly. If I save the buffer from your reproducer as a file and
later open in, the patch does solve the problem. However, when I run the
reproducer as is, a similar-looking problem emerges.

Using your reproducer, something very strange is going on.
With the patch, <delete> is correctly mapped to org-delete-char, which
calls (delete-char 1) at some point. However, this (delete-char 1)
somehow deletes 2!! chars instead of 1. This should be some tricky Emacs
issue (bug?). In any case, I am not ready to dig into what is going on
in Emacs. I do not think that such problem is going to appear during
normal Emacs usage. At least, I was unable to reproduce it by other
means than your reproducer function.

To illustrate the issue, one can use the following version of the
reproducer. Feel free to report this to Emacs upstream.

(defun emacs-bug-reproduce-45915 ()
  "Reproduce bug#45915."
  ;; (gnus-read-ephemeral-emacs-bug-group 45915)
  (interactive)
  (with-current-buffer (get-buffer-create "*temp buffer*")
    (erase-buffer)
    (require 'org)
    (orgtbl-mode 1)
    (insert (format "emacs-version: %s, org-version: %s\n" emacs-version org-version))
    (insert "|   | rownames |
| / | <l>      |
|---+----------|
|   | a        |"))
  (switch-to-buffer-other-window "*temp buffer*")
  (execute-kbd-macro (kbd "M-< C-n C-n C-n C-n C-f C-f"))
  (message "I will hit S-<return>.")
  (sit-for 2)
  (execute-kbd-macro (kbd "S-<return>"))
  (message "I will hit <delete>.")
  (sit-for 2)
  ;;  (execute-kbd-macro (kbd "<delete>"))
  (delete-char 1)
  (message "I will call previous-line.")
  (sit-for 2)
  (call-interactively 'previous-line))

Best,
Ihor

bug#45915: 28.1; delete-char deletes two letters

[Tak Kunihiro]

As Ihor pointed out, delete-char deletes two letters.

Can someone take a look and evaluate if this is issue with delete-char?
I simplified the recipes as shown below.

#+begin_src emacs-lisp
(defun emacs-bug-reproduce-45915 ()
  "Reproduce bug#45915."
  ;; (gnus-read-ephemeral-emacs-bug-group 45915)
  (interactive)
  (with-current-buffer (get-buffer-create "*temp buffer*")
    (erase-buffer)
    (require 'org)
    (orgtbl-mode 1)
    (insert (format "emacs-version: %s, org-version: %s\n" emacs-version org-version))
    (insert "| 1 |"))
  (switch-to-buffer-other-window "*temp buffer*")
  (execute-kbd-macro (kbd "M-< C-n C-f C-f"))
  (message "I will hit <S-return>.")
  (sit-for 2)
  (execute-kbd-macro (kbd "<S-return>"))
  (sit-for 2)
  (message "I will hit <delete>.")
  (sit-for 2)
  (execute-kbd-macro (kbd "<delete>"))
  (message "Two letters were deleted unexpectedly."))
;;; (call-interactively 'emacs-bug-reproduce-45915)
#+end_src

bug#45915: 28.2; delete-char deletes two letters

[Tak Kunihiro]

I confirmed that delete-char still deletes two letters on
emacs-version: 28.2, org-version: 9.5.5.

I found out (org-table-align) in (org-table-copy-down) makes
this situation.

Can someone take a look (org-table-align)?

TIA

#+begin_src emacs-lisp
(defun emacs-bug-reproduce-45915 ()
  "Reproduce bug#45915."
  ;; (gnus-read-ephemeral-emacs-bug-group 45915)
  (interactive)
  (with-current-buffer (get-buffer-create "*temp buffer*")
    (erase-buffer)
    (require 'org)
    (orgtbl-mode 1)
    (insert (format "emacs-version: %s, org-version: %s\n" emacs-version org-version))
    (insert "| 1 |"))
  (switch-to-buffer-other-window "*temp buffer*")
  (execute-kbd-macro (kbd "M-< C-n C-f C-f"))
  (message "I will call #'org-table-copy-down.")
  (sit-for 2)
  (call-interactively #'org-table-copy-down)
  (sit-for 2)
  (message "I will hit <delete>.")
  (sit-for 2)
  (execute-kbd-macro (kbd "<delete>"))
  (message "Two letters were deleted unexpectedly."))
;;; (call-interactively 'emacs-bug-reproduce-45915)
#+end_src

bug#45915: 28.2; delete-char deletes two letters

[Ihor Radchenko]

Tak Kunihiro <homeros.misasa@gmail.com> writes:

> I confirmed that delete-char still deletes two letters on
> emacs-version: 28.2, org-version: 9.5.5.
>
> I found out (org-table-align) in (org-table-copy-down) makes
> this situation.
>
> Can someone take a look (org-table-align)?

I think that the thread has been lost because the bug has been assigned
to Org. I now tried to reassign the bug back to emacs.

-- 
Ihor Radchenko,
Org mode contributor,
Learn more about Org mode at https://orgmode.org/.
Support Org development at https://liberapay.com/org-mode,
or support my work at https://liberapay.com/yantar92

bug#45915: 28.2; delete-char deletes two letters

[Tak Kunihiro]

I see a light.
With the following patch, <delete> deletes only a letter.

The (org-table-align) inserts string with text properties.
It seems that the text properties matter.

However, I do not understand what's going on there. Can you take a look
(org-table-align) with this hint and revise it in correct way?

diff --git a/lisp/org/org-table.el b/lisp/org/org-table.el
index 9b692d09736..55bc2838718 100644
--- a/lisp/org/org-table.el
+++ b/lisp/org/org-table.el
@@ -4410,6 +4410,7 @@ org-table-align
 					             alignments)
 				          "|")))
 		           "|")))
+              (setq new (substring-no-properties new))
 	      (if (equal new previous)
 		  (forward-line)
 		(insert new "\n")

bug#45915: 28.2; delete-char deletes two letters

[Eli Zaretskii]

> Cc: tkk@misasa.okayama-u.ac.jp, Ihor Radchenko <yantar92@gmail.com>
> From: Tak Kunihiro <homeros.misasa@gmail.com>
> Date: Sat, 17 Sep 2022 12:48:56 +0900
> 
> I see a light.
> With the following patch, <delete> deletes only a letter.
> 
> The (org-table-align) inserts string with text properties.
> It seems that the text properties matter.
> 
> However, I do not understand what's going on there. Can you take a look
> (org-table-align) with this hint and revise it in correct way?

Why do you think your proposed change is incorrect?

Rather than asking us to study how org-table works and why it
misbehaves in this case, would it be possible for some Org expert to
explain that in simple terms, so that the relation of what happens to
the relevant core Emacs features could be easier to realize?  For
example, what are the text properties involved in this case, the ones
that substring-no-properties removes?  Is it possible that one of
these properties is 'composition'?

bug#45915: 28.2; delete-char deletes two letters

[Tak Kunihiro]

Eli Zaretskii <eliz@gnu.org> writes:

>> Cc: tkk@misasa.okayama-u.ac.jp, Ihor Radchenko <yantar92@gmail.com>
>> From: Tak Kunihiro <homeros.misasa@gmail.com>
>> Date: Sat, 17 Sep 2022 12:48:56 +0900
>> 
>> I see a light.
>> With the following patch, <delete> deletes only a letter.
>> 
>> The (org-table-align) inserts string with text properties.
>> It seems that the text properties matter.
>> 
>> However, I do not understand what's going on there. Can you take a look
>> (org-table-align) with this hint and revise it in correct way?
>
> Why do you think your proposed change is incorrect?
>
> Rather than asking us to study how org-table works and why it
> misbehaves in this case, would it be possible for some Org expert to
> explain that in simple terms, so that the relation of what happens to
> the relevant core Emacs features could be easier to realize?  For
> example, what are the text properties involved in this case, the ones
> that substring-no-properties removes?  Is it possible that one of
> these properties is 'composition'?

Dear Eli, thank you for the response. I really want to fix it to
switch to 28.2 from 26.3!

1. Following function will create a problematic org-table.

(defun emacs-bug-reproduce-45915 ()
  "Reproduce bug#45915."
  (interactive)
  (with-current-buffer (get-buffer-create "*temp buffer*")
    (erase-buffer)
    (require 'org)
    (orgtbl-mode 1)
    (setq org-table-copy-increment nil)
    (insert (format "emacs-version: %s, org-version: %s\n" emacs-version org-version))
    (insert "| 1 |"))
  (switch-to-buffer-other-window "*temp buffer*")
  (execute-kbd-macro (kbd "M-< C-n C-f C-f"))
  (message "I will call #'org-table-copy-down.")
  (call-interactively #'org-table-copy-down))
;;; (call-interactively 'emacs-bug-reproduce-45915)

2. Buffer is with the 1x2 table as shown below. <delete> deletes one
letter on the first row, and it does two letters on the
second row.

  emacs-version: 28.2, org-version: 9.5.5
  | 1 |
  | 1 |

3. I wonder what is the difference between two lines.
I examined each line with following statement.

  (buffer-substring (line-beginning-position) (line-end-position))

line 1 returns: "| 1 |"
line 2 returns: #("| 1 |" 1 2 (display (space :relative-width 1)) 3 4
(display (space :relative-width 1)))

4. Although I do not know how to understand the return for line 2,
I speculated the problem is something to do with text properties.

5. It is (org-table-align) that generates and inserts the second line
with text properties.

6. I tried to remove text-properties from string to be inserted by
(org-table-align) using substring-no-properties (as shown in the
patch). Now, both lines returns the same "| 1 |". Problem disappeared.

7. Since I do not know intention why (org-table-align) wants to give
text properties to tables, I cannot tell the side effect of the patch.

TIA

bug#45915: 28.2; delete-char deletes two letters

[Eli Zaretskii]

> From: Tak Kunihiro <homeros.misasa@gmail.com>
> Cc: Tak Kunihiro <homeros.misasa@gmail.com>,  45915@debbugs.gnu.org,
>   yantar92@gmail.com
> Cc: tkk@misasa.okayama-u.ac.jp
> Date: Sat, 17 Sep 2022 17:20:26 +0900
> 
> 1. Following function will create a problematic org-table.
> 
> (defun emacs-bug-reproduce-45915 ()
>   "Reproduce bug#45915."
>   (interactive)
>   (with-current-buffer (get-buffer-create "*temp buffer*")
>     (erase-buffer)
>     (require 'org)
>     (orgtbl-mode 1)
>     (setq org-table-copy-increment nil)
>     (insert (format "emacs-version: %s, org-version: %s\n" emacs-version org-version))
>     (insert "| 1 |"))
>   (switch-to-buffer-other-window "*temp buffer*")
>   (execute-kbd-macro (kbd "M-< C-n C-f C-f"))
>   (message "I will call #'org-table-copy-down.")
>   (call-interactively #'org-table-copy-down))
> ;;; (call-interactively 'emacs-bug-reproduce-45915)
> 
> 2. Buffer is with the 1x2 table as shown below. <delete> deletes one
> letter on the first row, and it does two letters on the
> second row.
> 
>   emacs-version: 28.2, org-version: 9.5.5
>   | 1 |
>   | 1 |

No, it doesn't delete 2 characters in the 2nd line, it deletes just
one.  After you press <Delete>, type C-b (to go to the space between
the two '|' characters, and type "C-x =".  You will see this:

  Char: SPC (32, #o40, #x20, part of display "  ") point=51 of 54 (93%) column=1

Note the "51 to 54" part: there's a 'display' property there that
covers _two_ buffer positions, not one.  That's because the two
'display' properties on the two SPC characters before the deletion
have the same value:

>   (buffer-substring (line-beginning-position) (line-end-position))
> 
> line 1 returns: "| 1 |"
> line 2 returns: #("| 1 |" 1 2 (display (space :relative-width 1)) 3 4
> (display (space :relative-width 1)))

Those two 'display' properties are now displayed as a single stretch
glyph, because two adjacent text properties with the same value are
indistinguishable from a single one that spans all of the buffer
positions.

This is not a bug, this is how Emacs always worked.

The question is now why is this a problem for you, and how it affects
Org Table.  The solution, if there's a need for one, should IMO come
from the Org side.

bug#45915: 28.2; delete-char deletes two letters

[Tak Kunihiro]

>>   (buffer-substring (line-beginning-position) (line-end-position))
>> 
>> line 1 returns: "| 1 |"
>> line 2 returns: #("| 1 |" 1 2 (display (space :relative-width 1)) 3 4
>> (display (space :relative-width 1)))
>
> Those two 'display' properties are now displayed as a single stretch
> glyph, because two adjacent text properties with the same value are
> indistinguishable from a single one that spans all of the buffer
> positions.
>
> This is not a bug, this is how Emacs always worked.

I see. I understand that 'display' properties on an org-table matters.
I took a look for (org-table-align) and found that
`org-table-separator-space' defines the text properties as shown below.

org-table.el: (defconst org-table-separator-space
org-table.el:   (propertize " " 'display '(space :relative-width 1))
org-table.el:   "Space used around fields when aligning the table.
org-table.el: This space serves as a segment separator for the purposes of the
org-table.el: bidirectional reordering.")

Although I still do not understand the purpose of `:relative-width',
problem disappeared with following configuration.

  (with-eval-after-load "org-table"
    (setq org-table-separator-space " "))

I have an impression that default characters on
org-table-separator-space is problematic.

bug#45915: 28.2; delete-char deletes two letters

[Eli Zaretskii]

> From: Tak Kunihiro <homeros.misasa@gmail.com>
> Cc: tkk@misasa.okayama-u.ac.jp
> Date: Sun, 18 Sep 2022 14:44:59 +0900
> 
> I see. I understand that 'display' properties on an org-table matters.
> I took a look for (org-table-align) and found that
> `org-table-separator-space' defines the text properties as shown below.
> 
> org-table.el: (defconst org-table-separator-space
> org-table.el:   (propertize " " 'display '(space :relative-width 1))
> org-table.el:   "Space used around fields when aligning the table.
> org-table.el: This space serves as a segment separator for the purposes of the
> org-table.el: bidirectional reordering.")

Yes, and that's exactly right for when the table could include
bidirectional (i.e. mixed L2R and R2L) text.

> Although I still do not understand the purpose of `:relative-width',
> problem disappeared with following configuration.
> 
>   (with-eval-after-load "org-table"
>     (setq org-table-separator-space " "))

I don't think this is the right solution.  One possible solution that
is better is to bind <Delete> to an org-table specific command, which
would insert some character displayed as a thin space between the two
spaces that have the 'display' property.  But I'll let Ihor chime in.

> I have an impression that default characters on
> org-table-separator-space is problematic.

What do you mean by "default characters"?

bug#45915: 28.2; delete-char deletes two letters

[Eli Zaretskii]

> Cc: tkk@misasa.okayama-u.ac.jp, yantar92@gmail.com, 45915@debbugs.gnu.org
> Date: Sun, 18 Sep 2022 08:56:48 +0300
> From: Eli Zaretskii <eliz@gnu.org>
> 
> I don't think this is the right solution.  One possible solution that
> is better is to bind <Delete> to an org-table specific command, which
> would insert some character displayed as a thin space between the two
> spaces that have the 'display' property.  But I'll let Ihor chime in.

One other possibility is to use a slightly different :relative-width
factor for the two spaces in a table cell: one with the value of 1,
the other with 1.001 (say).  They will be indistinguishable on
display, but since the values are not equal, both stretch gfyphs will
be displayed, not just one.

bug#45915: 28.2; delete-char deletes two letters

[Tak Kunihiro]

> One other possibility is to use a slightly different :relative-width
> factor for the two spaces in a table cell: one with the value of 1,
> the other with 1.001 (say).  They will be indistinguishable on
> display, but since the values are not equal, both stretch gfyphs will
> be displayed, not just one.

Function (org-table--align-field) in org-table.el sandwiches 
text by org-table-separator-space as shown below.

    (concat org-table-separator-space
 	    prefix
 	    field
 	    suffix
-	    org-table-separator-space)))
+	    (propertize org-table-separator-space 'display '(space :relative-width 1.001)))))

I confirmed that this modification works good.

>> I have an impression that default characters on
>> org-table-separator-space is problematic.
>
> What do you mean by "default characters"?

Let me take the statement back.

bug#45915: 28.2; delete-char deletes two letters

[Ihor Radchenko]

Eli Zaretskii <eliz@gnu.org> writes:

>> I don't think this is the right solution.  One possible solution that
>> is better is to bind <Delete> to an org-table specific command, which
>> would insert some character displayed as a thin space between the two
>> spaces that have the 'display' property.  But I'll let Ihor chime in.
>
> One other possibility is to use a slightly different :relative-width
> factor for the two spaces in a table cell: one with the value of 1,
> the other with 1.001 (say).  They will be indistinguishable on
> display, but since the values are not equal, both stretch gfyphs will
> be displayed, not just one.

This is a good idea. I used it to fix the reported issue.

Fixed on main.
https://git.savannah.gnu.org/cgit/emacs/org-mode.git/commit/?id=9dde82411aefe5a6728eef95c3629dbf266fefa5

-- 
Ihor Radchenko,
Org mode contributor,
Learn more about Org mode at https://orgmode.org/.
Support Org development at https://liberapay.com/org-mode,
or support my work at https://liberapay.com/yantar92

Re: bug#45915: 28.2; delete-char deletes two letters

[Ihor Radchenko]

Eli Zaretskii <eliz@gnu.org> writes:

>> I don't think this is the right solution.  One possible solution that
>> is better is to bind <Delete> to an org-table specific command, which
>> would insert some character displayed as a thin space between the two
>> spaces that have the 'display' property.  But I'll let Ihor chime in.
>
> One other possibility is to use a slightly different :relative-width
> factor for the two spaces in a table cell: one with the value of 1,
> the other with 1.001 (say).  They will be indistinguishable on
> display, but since the values are not equal, both stretch gfyphs will
> be displayed, not just one.

This is a good idea. I used it to fix the reported issue.

Fixed on main.
https://git.savannah.gnu.org/cgit/emacs/org-mode.git/commit/?id=9dde82411aefe5a6728eef95c3629dbf266fefa5

-- 
Ihor Radchenko,
Org mode contributor,
Learn more about Org mode at https://orgmode.org/.
Support Org development at https://liberapay.com/org-mode,
or support my work at https://liberapay.com/yantar92

bug#45915: 28.2; delete-char deletes two letters

[Eli Zaretskii]

> From: Ihor Radchenko <yantar92@gmail.com>
> Cc: homeros.misasa@gmail.com,  tkk@misasa.okayama-u.ac.jp,
>   45915@debbugs.gnu.org, emacs-orgmode@gnu.org
> Date: Thu, 22 Sep 2022 20:03:55 +0800
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > One other possibility is to use a slightly different :relative-width
> > factor for the two spaces in a table cell: one with the value of 1,
> > the other with 1.001 (say).  They will be indistinguishable on
> > display, but since the values are not equal, both stretch gfyphs will
> > be displayed, not just one.
> 
> This is a good idea. I used it to fix the reported issue.
> 
> Fixed on main.

Great.  So I guess this bug report can be closed?

Re: bug#45915: 28.2; delete-char deletes two letters

[Eli Zaretskii]

> From: Ihor Radchenko <yantar92@gmail.com>
> Cc: homeros.misasa@gmail.com,  tkk@misasa.okayama-u.ac.jp,
>   45915@debbugs.gnu.org, emacs-orgmode@gnu.org
> Date: Thu, 22 Sep 2022 20:03:55 +0800
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > One other possibility is to use a slightly different :relative-width
> > factor for the two spaces in a table cell: one with the value of 1,
> > the other with 1.001 (say).  They will be indistinguishable on
> > display, but since the values are not equal, both stretch gfyphs will
> > be displayed, not just one.
> 
> This is a good idea. I used it to fix the reported issue.
> 
> Fixed on main.

Great.  So I guess this bug report can be closed?

bug#45915: 28.2; delete-char deletes two letters

[Tak Kunihiro]

>>> One other possibility is to use a slightly different :relative-width
>>> factor for the two spaces in a table cell: one with the value of 1,
>>> the other with 1.001 (say).  They will be indistinguishable on
>>> display, but since the values are not equal, both stretch gfyphs will
>>> be displayed, not just one.
>> 
>> This is a good idea. I used it to fix the reported issue.
>> 
>> Fixed on main.
> 
> Great.  So I guess this bug report can be closed?

Nice. Yes, please close this bug report.

Re: bug#45915: 28.2; delete-char deletes two letters

[Tak Kunihiro]

>>> One other possibility is to use a slightly different :relative-width
>>> factor for the two spaces in a table cell: one with the value of 1,
>>> the other with 1.001 (say).  They will be indistinguishable on
>>> display, but since the values are not equal, both stretch gfyphs will
>>> be displayed, not just one.
>> 
>> This is a good idea. I used it to fix the reported issue.
>> 
>> Fixed on main.
> 
> Great.  So I guess this bug report can be closed?

Nice. Yes, please close this bug report.

bug#45915: 28.2; delete-char deletes two letters

[Eli Zaretskii]

> Date: Fri, 23 Sep 2022 06:36:58 +0900 (JST)
> Cc: yantar92@gmail.com, homeros.misasa@gmail.com, 45915@debbugs.gnu.org,
>  emacs-orgmode@gnu.org, tkk@misasa.okayama-u.ac.jp
> From: Tak Kunihiro <tkk@misasa.okayama-u.ac.jp>
> 
> >>> One other possibility is to use a slightly different :relative-width
> >>> factor for the two spaces in a table cell: one with the value of 1,
> >>> the other with 1.001 (say).  They will be indistinguishable on
> >>> display, but since the values are not equal, both stretch gfyphs will
> >>> be displayed, not just one.
> >> 
> >> This is a good idea. I used it to fix the reported issue.
> >> 
> >> Fixed on main.
> > 
> > Great.  So I guess this bug report can be closed?
> 
> Nice. Yes, please close this bug report.

Thanks, done.

Re: bug#45915: 28.2; delete-char deletes two letters

[Eli Zaretskii]

> Date: Fri, 23 Sep 2022 06:36:58 +0900 (JST)
> Cc: yantar92@gmail.com, homeros.misasa@gmail.com, 45915@debbugs.gnu.org,
>  emacs-orgmode@gnu.org, tkk@misasa.okayama-u.ac.jp
> From: Tak Kunihiro <tkk@misasa.okayama-u.ac.jp>
> 
> >>> One other possibility is to use a slightly different :relative-width
> >>> factor for the two spaces in a table cell: one with the value of 1,
> >>> the other with 1.001 (say).  They will be indistinguishable on
> >>> display, but since the values are not equal, both stretch gfyphs will
> >>> be displayed, not just one.
> >> 
> >> This is a good idea. I used it to fix the reported issue.
> >> 
> >> Fixed on main.
> > 
> > Great.  So I guess this bug report can be closed?
> 
> Nice. Yes, please close this bug report.

Thanks, done.

bug#45915: 28.2; delete-char deletes two letters

[Tak Kunihiro]

OK. Now I understand <delete> only deletes one letter.  Two spaces are
shown as one letter, as expected.

To have a distorted table soon after deletion of text is OK.

However, when two spaces are shown as one letter, point moves randomly
when I press <up>.  I cannot keep editing with this condition.

I show recipe to reproduce this situation. Is there a trick to move
point in expected way?

(defun emacs-bug-reproduce-45915 ()
  "Reproduce bug#45915."
  ;; (gnus-read-ephemeral-emacs-bug-group 45915)
  (interactive)
  (with-current-buffer (get-buffer-create "*temp buffer*")
    (erase-buffer)
    (require 'org)
    (orgtbl-mode 1)
    (insert (format "emacs-version: %s, org-version: %s\n" emacs-version org-version))
    (insert "| 1 |   |   |   |   |   |   |   |   |"))
  (switch-to-buffer-other-window "*temp buffer*")
  (execute-kbd-macro (kbd "M-< C-n C-f C-f"))
  (message "I will call #'org-table-copy-down.")
  (sit-for 2)
  (call-interactively #'org-table-copy-down)
  (sit-for 2)
  (message "I will hit <delete>.")
  (sit-for 2)
  (execute-kbd-macro (kbd "<delete>"))
  (message "One letter was deleted expectedly.")
  (sit-for 2)
  (message "I will hit <up>.")
  (sit-for 2)
  (execute-kbd-macro (kbd "<up>"))
  (message "Point moved unexpectedly."))
;;; (call-interactively 'emacs-bug-reproduce-45915)

bug#45915: 28.2; delete-char deletes two letters

[Eli Zaretskii]

> From: Tak Kunihiro <homeros.misasa@gmail.com>
> Cc: Tak Kunihiro <homeros.misasa@gmail.com>,  yantar92@gmail.com,
>   45915@debbugs.gnu.org
> Cc: tkk@misasa.okayama-u.ac.jp
> Date: Mon, 19 Sep 2022 10:24:55 +0900
> 
> However, when two spaces are shown as one letter, point moves randomly
> when I press <up>.  I cannot keep editing with this condition.
> 
> I show recipe to reproduce this situation. Is there a trick to move
> point in expected way?

This is an unrelated bug, now fixed on the master branch.

(A workaround is to press C-f or <RIGHT> before <UP>.)