bug#58928: 29.0.50; overlays in org-mode are disrupted after call `org-capture`

bug#58928: 29.0.50; overlays in org-mode are disrupted after call `org-capture`

[Eli Qian]

Reproduction steps:

1. start "emacs -Q"
2. type "C-x C-f" and "test.org" and RET
3. type the following:
* DONE test
CLOSED: [2022-10-31 Mon 18:27]
:LOGBOOK:
- State "DONE"       from "TODO"       [2022-10-31 Mon 18:27]
:END:
* DONE test
CLOSED: [2022-10-31 Mon 18:27]
:LOGBOOK:
- State "DONE"       from "TODO"       [2022-10-31 Mon 18:27]
:END:
* DONE test
CLOSED: [2022-10-31 Mon 18:27]
:LOGBOOK:
- State "DONE"       from "TODO"       [2022-10-31 Mon 18:27]
:END:
* DONE test
CLOSED: [2022-10-31 Mon 18:27]
:LOGBOOK:
- State "DONE"       from "TODO"       [2022-10-31 Mon 18:27]
:END:

4. call `org-shifttab`(S-TAB), make sure all headlines are in 
OVERVIEW.
5. load the following code:
(setq org-capture-templates
      '(("l" "test" entry (file "path to org")
           "* %?"
           :prepend t
           :empty-lines 0)))

6. Call `org-capture` in test.org buffer, and type something.


Actual:

You will see overlays (org-ellipsis) will move randomly, and 
reopening this file will return to normal.

Expected:

`org-capture` works fine, no format disruption.


In GNU Emacs 29.0.50 (build 1, x86_64-pc-linux-gnu, GTK+ Version
 3.24.34, cairo version 1.17.6) of 2022-10-31 built on eli
Repository revision: a691e811e23dee48674e9e0716429074fdbe3d7b
Repository branch: master
Windowing system distributor 'The X.Org Foundation', version 
11.0.12101004
System Description: Arch Linux

Configured using:
 'configure --with-mailutils --with-native-compilation 
 --with-modules
 --with-xwidgets'

Configured features:
ACL CAIRO DBUS FREETYPE GIF GLIB GMP GNUTLS GPM GSETTINGS HARFBUZZ 
JPEG
JSON LCMS2 LIBSELINUX LIBSYSTEMD LIBXML2 MODULES NATIVE_COMP 
NOTIFY
INOTIFY PDUMPER PNG RSVG SECCOMP SOUND SQLITE3 THREADS TIFF
TOOLKIT_SCROLL_BARS WEBP X11 XDBE XIM XINPUT2 XPM XWIDGETS GTK3 
ZLIB

Important settings:
  value of $LANG: en_US.UTF-8
  locale-coding-system: utf-8-unix

Major mode: Org

Minor modes in effect:
  tooltip-mode: t
  global-eldoc-mode: t
  show-paren-mode: t
  electric-indent-mode: t
  mouse-wheel-mode: t
  tool-bar-mode: t
  menu-bar-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  blink-cursor-mode: t
  line-number-mode: t
  indent-tabs-mode: t
  transient-mark-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t

Load-path shadows:
None found.

Features:
(mule-util info pp shadow sort mail-extr emacsbug vc-git diff-mode
easy-mmode vc-dispatcher org-element avl-tree generator ol-eww eww 
xdg
url-queue thingatpt mm-url ol-rmail ol-mhe ol-irc ol-info ol-gnus
nnselect gnus-art mm-uu mml2015 mm-view mml-smime smime gnutls dig
gnus-sum shr pixel-fill kinsoku url-file svg dom browse-url url
url-proxy url-privacy url-expand url-methods url-history 
url-cookie
generate-lisp-file url-domsuf url-util url-parse auth-source eieio
eieio-core url-vars gnus-group gnus-undo gnus-start gnus-dbus dbus 
xml
gnus-cloud nnimap nnmail mail-source utf7 nnoo parse-time 
gnus-spec
gnus-int gnus-range message sendmail mailcap yank-media puny 
rfc822 mml
mml-sec password-cache epa derived epg rfc6068 epg-config 
mm-decode
mm-bodies mm-encode mail-parse rfc2231 rfc2047 rfc2045 ietf-drums
mailabbrev gmm-utils mailheader gnus-win gnus nnheader gnus-util
text-property-search mail-utils range mm-util mail-prsvr wid-edit
ol-docview doc-view filenotify jka-compr image-mode exif dired
dired-loaddefs ol-bibtex ol-bbdb ol-w3m ol-doi org-link-doi org ob
ob-tangle ob-ref ob-lob ob-table ob-exp org-macro org-footnote 
org-src
ob-comint org-pcomplete pcomplete comint ansi-osc ansi-color ring
org-list org-faces org-entities noutline outline org-version
ob-emacs-lisp ob-core ob-eval org-table oc-basic json map bibtex 
iso8601
time-date ol org-keys oc org-compat byte-opt advice org-macs
org-loaddefs format-spec find-func cal-menu calendar cal-loaddefs
cl-loaddefs comp comp-cstr warnings icons subr-x rx cl-seq cl-macs 
gv
cl-extra help-mode bytecomp byte-compile cl-lib rmc iso-transl 
tooltip
cconv eldoc paren electric uniquify ediff-hook vc-hooks 
lisp-float-type
elisp-mode mwheel term/x-win x-win term/common-win x-dnd tool-bar 
dnd
fontset image regexp-opt fringe tabulated-list replace newcomment
text-mode lisp-mode prog-mode register page tab-bar menu-bar 
rfn-eshadow
isearch easymenu timer select scroll-bar mouse jit-lock font-lock 
syntax
font-core term/tty-colors frame minibuffer nadvice seq simple 
cl-generic
indonesian philippine cham georgian utf-8-lang misc-lang 
vietnamese
tibetan thai tai-viet lao korean japanese eucjp-ms cp51932 hebrew 
greek
romanian slovak czech european ethiopic indian cyrillic chinese
composite emoji-zwj charscript charprop case-table epa-hook
jka-cmpr-hook help abbrev obarray oclosure cl-preloaded button 
loaddefs
theme-loaddefs faces cus-face macroexp files window 
text-properties
overlay sha1 md5 base64 format env code-pages mule custom widget 
keymap
hashtable-print-readable backquote threads xwidget-internal 
dbusbind
inotify lcms2 dynamic-setting system-font-setting 
font-render-setting
cairo move-toolbar gtk x-toolkit xinput2 x multi-tty
make-network-process native-compile emacs)

Memory information:
((conses 16 265635 19353)
 (symbols 48 20861 6)
 (strings 32 75783 8754)
 (string-bytes 1 2324175)
 (vectors 16 40494)
 (vector-slots 8 716446 33750)
 (floats 8 354 360)
 (intervals 56 1052 21)
 (buffers 984 15))
 
-- 

bug#58928: 29.0.50; overlays in org-mode are disrupted after call `org-capture`

[Eli Zaretskii]

> 1. start "emacs -Q"
> 2. type "C-x C-f" and "test.org" and RET
> 3. type the following:
> * DONE test
> CLOSED: [2022-10-31 Mon 18:27]
> :LOGBOOK:
> - State "DONE"       from "TODO"       [2022-10-31 Mon 18:27]
> :END:
> * DONE test
> CLOSED: [2022-10-31 Mon 18:27]
> :LOGBOOK:
> - State "DONE"       from "TODO"       [2022-10-31 Mon 18:27]
> :END:
> * DONE test
> CLOSED: [2022-10-31 Mon 18:27]
> :LOGBOOK:
> - State "DONE"       from "TODO"       [2022-10-31 Mon 18:27]
> :END:
> * DONE test
> CLOSED: [2022-10-31 Mon 18:27]
> :LOGBOOK:
> - State "DONE"       from "TODO"       [2022-10-31 Mon 18:27]
> :END:
> 
> 4. call `org-shifttab`(S-TAB), make sure all headlines are in 
> OVERVIEW.
> 5. load the following code:
> (setq org-capture-templates
>       '(("l" "test" entry (file "path to org")
>            "* %?"
>            :prepend t
>            :empty-lines 0)))
> 
> 6. Call `org-capture` in test.org buffer, and type something.
> 
> 
> Actual:
> 
> You will see overlays (org-ellipsis) will move randomly, and 
> reopening this file will return to normal.
> 
> Expected:
> 
> `org-capture` works fine, no format disruption.

Could someone please look at this problem ASAP?

Thanks.

bug#58928: 29.0.50; overlays in org-mode are disrupted after call `org-capture`

[Gerd Möllmann]

On 01.11.22 07:14, Eli Zaretskii wrote:
>> 1. start "emacs -Q"
>> 2. type "C-x C-f" and "test.org" and RET
>> 3. type the following:
>> * DONE test
>> CLOSED: [2022-10-31 Mon 18:27]
>> :LOGBOOK:
>> - State "DONE"       from "TODO"       [2022-10-31 Mon 18:27]
>> :END:
>> * DONE test
>> CLOSED: [2022-10-31 Mon 18:27]
>> :LOGBOOK:
>> - State "DONE"       from "TODO"       [2022-10-31 Mon 18:27]
>> :END:
>> * DONE test
>> CLOSED: [2022-10-31 Mon 18:27]
>> :LOGBOOK:
>> - State "DONE"       from "TODO"       [2022-10-31 Mon 18:27]
>> :END:
>> * DONE test
>> CLOSED: [2022-10-31 Mon 18:27]
>> :LOGBOOK:
>> - State "DONE"       from "TODO"       [2022-10-31 Mon 18:27]
>> :END:
>>
>> 4. call `org-shifttab`(S-TAB), make sure all headlines are in
>> OVERVIEW.
>> 5. load the following code:
>> (setq org-capture-templates
>>        '(("l" "test" entry (file "path to org")
>>             "* %?"
>>             :prepend t
>>             :empty-lines 0)))

I'm not using this part of Org, and need some hand-holding here.

I've evaluated the form above as-is in *scratch*, and done the S-tab.
I now have 4 lines looking like "* DONE test..." in the buffer.

>>
>> 6. Call `org-capture` in test.org buffer, and type something.

M-x org-capture RET in test.org asks me what template to use, and I
choose 'l'.  I then type something and hit C-c C-c.  It asks me to let
it create a directory ~/org, and I let it do that.  What I see after
that in the test.log buffer looks like it did before.  There are same 4
lines.

What am I doing wrong?

(This is current master.)

bug#58928: 29.0.50; overlays in org-mode are disrupted after call `org-capture`

[Gerd Möllmann]

On 01.11.22 08:11, Eli Qian wrote:
> This is my first bug reporting, so I apologize if there are any omissions.
> 
> On Tue, Nov 01 2022, Gerd Möllmann <gerd.moellmann@gmail.com> wrote:
> 
>> I've evaluated the form above as-is in *scratch*, and done the S-tab.
>> I now have 4 lines looking like "* DONE test..." in the buffer.
> 
> After Step 4, save the buffer to "~/test.org". Now You have 4 lines looking
> like "* DONE test..." in "~/test.org".
> 
> In Step 5, the actual code is
> 
> (setq org-capture-templates
>       '(("l" "global notes" entry (file "~/test.org")
>            "* %?"
>            :prepend t
>            :empty-lines 0)))
>>>> 6. Call `org-capture` in test.org buffer, and type something.
>>
>> M-x org-capture RET in test.org asks me what template to use, and I
>> choose 'l'.  I then type something and hit C-c C-c.  It asks me to let
>> it create a directory ~/org, and I let it do that.  What I see after
>> that in the test.log buffer looks like it did before.  There are same 4
>> lines.
>>
>> What am I doing wrong?
>>
>> (This is current master.)
> 
> Step 6:
> M-x org-capture RET in test.org , you will see the capture buffer appears
> on the right, while test.org buffer on the left. Then type something,
> you will see the text in test.org buffer disrupted.

Thanks, that worked.  (It's not a big deal, but please keep the others
in the CC.)

What I see now is that I have 2 windows.  One is showing test.org, the
other is a buffer CAPTURE-test.org.  While typing in CAPTURE-test.org,
the text in test.org is mangled with each key stroke.  The invisible
parts (...) are moved around in test.org.  C-l doesn't change anything.

Running with --enable-checking and ASAN doesn't show anything.

Do we have someone who knows Org internals, and can tell what exactly 
typing in CAPTURE-test.org does, presumably with overlays?

bug#58928: 29.0.50; overlays in org-mode are disrupted after call `org-capture`

[Gerd Möllmann]

Gerd Möllmann <gerd.moellmann@gmail.com> writes:

> Running with --enable-checking and ASAN doesn't show anything.

Make check also doesn't show anything.

The manual tests in test/manual/noverlay are apparently out of sync with
src/itree.[ch], they don't compile.

bug#58928: 29.0.50; overlays in org-mode are disrupted after call `org-capture`

[Ihor Radchenko]

Gerd Möllmann <gerd.moellmann@gmail.com> writes:

> What I see now is that I have 2 windows.  One is showing test.org, the
> other is a buffer CAPTURE-test.org.  While typing in CAPTURE-test.org,
> the text in test.org is mangled with each key stroke.  The invisible
> parts (...) are moved around in test.org.  C-l doesn't change anything.
>
> Running with --enable-checking and ASAN doesn't show anything.
>
> Do we have someone who knows Org internals, and can tell what exactly 
> typing in CAPTURE-test.org does, presumably with overlays?

I cannot reproduce using somewhat old Emacs 29
Development version 41d3d71a1e67 on master branch; build date 2022-10-28.

I can reproduce after updating to the latest master.

FYI, CAPTURE-test.org is nothing but a narrowed indirect buffer.
In this particular case, CAPTURE-test.org is narrowed to 1..4 before you
start typing. You can widen it and will see that it is still the
original Org file.

Note that test.org itself is also narrowed, which should not happen.
Although, narrowing of test.org during capture can also be seen using
Emacs 28.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

bug#58928: 29.0.50; overlays in org-mode are disrupted after call `org-capture`

[Gerd Möllmann]

On 01.11.22 10:07, Ihor Radchenko wrote:
> Gerd Möllmann <gerd.moellmann@gmail.com> writes:
> 
>> What I see now is that I have 2 windows.  One is showing test.org, the
>> other is a buffer CAPTURE-test.org.  While typing in CAPTURE-test.org,
>> the text in test.org is mangled with each key stroke.  The invisible
>> parts (...) are moved around in test.org.  C-l doesn't change anything.
>>
>> Running with --enable-checking and ASAN doesn't show anything.
>>
>> Do we have someone who knows Org internals, and can tell what exactly
>> typing in CAPTURE-test.org does, presumably with overlays?
> 
> I cannot reproduce using somewhat old Emacs 29
> Development version 41d3d71a1e67 on master branch; build date 2022-10-28.
> 
> I can reproduce after updating to the latest master.
> 
> FYI, CAPTURE-test.org is nothing but a narrowed indirect buffer.
> In this particular case, CAPTURE-test.org is narrowed to 1..4 before you
> start typing. You can widen it and will see that it is still the
> original Org file.
> 
> Note that test.org itself is also narrowed, which should not happen.
> Although, narrowing of test.org during capture can also be seen using
> Emacs 28.
> 

Thanks for looking into this, Ihor!

I don't know, but my guess is that the indirect buffer thing is already
a good starting point for Stefan/Matt.

bug#58928: 29.0.50; overlays in org-mode are disrupted after call `org-capture`

[Matt Armstrong]

> Thanks for looking into this, Ihor!
>
> I don't know, but my guess is that the indirect buffer thing is
> already a good starting point for Stefan/Matt.

I can repro this with a test.org file containing one simple entry:

----------------------------------------------------------------------
* DONE test
My dog has fleas, a *lot* of fleas!
----------------------------------------------------------------------

As Eli Quian instructed, use Shift-TAB in test.org to get it like this:

----------------------------------------------------------------------
* DONE test...
----------------------------------------------------------------------

At that point there is one overlay:

----------------------------------------------------------------------
Buffer #<buffer test.org> has 1 overlays.

#<overlay from 12 to 49 in test.org>
(isearch-open-invisible
 #[128 "\300\301!\207"
       [org-show-context isearch]
       3 "\n\n(fn &rest _)"]
 invisible outline evaporate t)
----------------------------------------------------------------------

This covers from the newline just after "* DONE test" to end of buffer.

Now, "M-x org-capture" gets you a capture buffer and the test.org buffer
is already displaying the wrong thing:

----------------------------------------------------------------------
* DONE t...s!
----------------------------------------------------------------------

while CAPTURE-test.org is displaying this (the '|' is (point)):

----------------------------------------------------------------------
* |
----------------------------------------------------------------------

If I erase the "* " from CAPTURE-test.org then test.org displays this:

----------------------------------------------------------------------
* DONE tes...
----------------------------------------------------------------------

And if I "M-x widen" CAPTURE-test.org I see this:

----------------------------------------------------------------------

* DONE test
My dog has fleas, a *lot* of fleas!
----------------------------------------------------------------------

I cancel out and re-do the M-x org-capture, then type in a new org entry
in CAPTURE-test.org, which looks like:

----------------------------------------------------------------------
* My new org heading
----------------------------------------------------------------------

...and test.org looks like this:

----------------------------------------------------------------------
...s, a *lot* of fleas!
----------------------------------------------------------------------

bug#58928: 29.0.50; overlays in org-mode are disrupted after call `org-capture`

[Matt Armstrong]

[-- Attachment #1: Type: text/plain, Size: 1025 bytes --]

I've created a simpler repro that demonstrates how inserts/deletes in an
indirect buffer does not update positions for overlays its base buffer.

This function creates a temp buffer with some text and one overlay, then
an indirect buffer against it.  Editing text in the indirect buffer
fails to update overlay positions in the base buffer.

(defun repro58928 ()
  (interactive)
  (with-current-buffer (get-buffer-create "*b58928*")
    (erase-buffer)
    (delete-all-overlays)
    (insert "My dog has fleas,\nlots and lots of fleas!\n")
    (let ((overlay (make-overlay 18 (point-max))))
      (overlay-put overlay 'face 'match))
    (pop-to-buffer (current-buffer))
    (ignore-errors
      (kill-buffer "*INDIRECT-b58928*"))
    (let ((indirect (make-indirect-buffer (current-buffer)
					  "*INDIRECT-b58928*")))
      (with-current-buffer indirect
	(goto-char (point-min)))
      (pop-to-buffer indirect))))

I'm not sure where the bug is, but the good news is that the regression
test will be easy to express in lisp.


[-- Attachment #2: before typing in the indirect buffer --]
[-- Type: image/png, Size: 34718 bytes --]

[-- Attachment #3: after typing in the indirect buffer --]
[-- Type: image/png, Size: 40692 bytes --]

bug#58928: 29.0.50; overlays in org-mode are disrupted after call `org-capture`

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

> I don't know, but my guess is that the indirect buffer thing is already
> a good starting point for Stefan/Matt.

Indeed.  I believe it is now fixed.  Please confirm,


        Stefan

bug#58928: 29.0.50; overlays in org-mode are disrupted after call `org-capture`

[Matt Armstrong]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> I don't know, but my guess is that the indirect buffer thing is already
>> a good starting point for Stefan/Matt.
>
> Indeed.  I believe it is now fixed.  Please confirm,

Fix confirmed on my end.

This bug made me realize indirect buffers have another failure mode,
which turned out to be a bug.  Because this bug seems to exist prior to
the noverlay work (i.e. it fails in Emacs 27 too) I filed it as a
separate bug:

bug#58958: Acknowledgement (29.0.50; overlays fail to evaporate across
indirect buffers)

bug#58928: 29.0.50; overlays in org-mode are disrupted after call `org-capture`

[Gerd Möllmann]

On 02.11.22 02:39, Stefan Monnier wrote:
>> I don't know, but my guess is that the indirect buffer thing is already
>> a good starting point for Stefan/Matt.
> 
> Indeed.  I believe it is now fixed.  Please confirm,

Works for me.  Thanks!

bug#58928: 29.0.50; overlays in org-mode are disrupted after call `org-capture`

[Eason Huang]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> I don't know, but my guess is that the indirect buffer thing is already
>> a good starting point for Stefan/Matt.
>
> Indeed.  I believe it is now fixed.  Please confirm,

Hi stefan,

I tried the latest commit, but thhis issue is still existed on macOS and
Windows 10.

I tested commit 8a5678906fa1b899c4d111e5ee4334b278f50d48 on Windows 10,
and commit 2eb2eb0c9ead205935be58ce9bb54465d48b20c8 on macOS Ventura
(13.0)


#-------------------------------------------------------------------
In GNU Emacs 29.0.50 (build 1, x86_64-apple-darwin22.1.0, NS
 appkit-2299.00 Version 13.0 (Build 22A380)) of 2022-11-03 built on
 MacBook
Repository revision: 2eb2eb0c9ead205935be58ce9bb54465d48b20c8
Repository branch: master
Windowing system distributor 'Apple', version 10.3.2299
System Description:  macOS 13.0

Configured using:
 'configure --with-native-compilation=aot --without-dbus
 'CPPFLAGS=-I/opt/local/include
 -isysroot/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk'
 'LDFLAGS=-L/opt/local/lib -Wl,-headerpad_max_install_names -Wl,-rpath
 /opt/local/lib/gcc12
 -Wl,-syslibroot,/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk
 -arch x86_64''

Configured features:
ACL GIF GLIB GMP GNUTLS JPEG JSON LCMS2 LIBXML2 MODULES NATIVE_COMP
NOTIFY KQUEUE NS PDUMPER PNG RSVG SQLITE3 THREADS TIFF
TOOLKIT_SCROLL_BARS WEBP XIM ZLIB

Important settings:
  value of $LC_CTYPE: en_US.UTF-8
  value of $LANG: en_US.UTF-8
  locale-coding-system: utf-8-unix

Major mode: Lisp Interaction

Minor modes in effect:
  tooltip-mode: t
  global-eldoc-mode: t
  eldoc-mode: t
  show-paren-mode: t
  electric-indent-mode: t
  mouse-wheel-mode: t
  tool-bar-mode: t
  menu-bar-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  blink-cursor-mode: t
  line-number-mode: t
  indent-tabs-mode: t
  transient-mark-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t

Load-path shadows:
None found.

Features:
(shadow sort mail-extr emacsbug message mailcap yank-media puny dired
dired-loaddefs rfc822 mml mml-sec password-cache epa derived epg rfc6068
epg-config gnus-util text-property-search mm-decode mm-bodies mm-encode
mail-parse rfc2231 mailabbrev gmm-utils mailheader sendmail rfc2047
rfc2045 ietf-drums mm-util mail-prsvr mail-utils time-date vc-git
diff-mode easy-mmode vc-dispatcher cl-loaddefs comp comp-cstr warnings
icons subr-x rx cl-seq cl-macs gv cl-extra help-mode bytecomp
byte-compile cl-lib rmc iso-transl tooltip cconv eldoc paren electric
uniquify ediff-hook vc-hooks lisp-float-type elisp-mode mwheel
term/ns-win ns-win ucs-normalize mule-util term/common-win tool-bar dnd
fontset image regexp-opt fringe tabulated-list replace newcomment
text-mode lisp-mode prog-mode register page tab-bar menu-bar rfn-eshadow
isearch easymenu timer select scroll-bar mouse jit-lock font-lock syntax
font-core term/tty-colors frame minibuffer nadvice seq simple cl-generic
indonesian philippine cham georgian utf-8-lang misc-lang vietnamese
tibetan thai tai-viet lao korean japanese eucjp-ms cp51932 hebrew greek
romanian slovak czech european ethiopic indian cyrillic chinese
composite emoji-zwj charscript charprop case-table epa-hook
jka-cmpr-hook help abbrev obarray oclosure cl-preloaded button loaddefs
theme-loaddefs faces cus-face macroexp files window text-properties
overlay sha1 md5 base64 format env code-pages mule custom widget keymap
hashtable-print-readable backquote threads kqueue cocoa ns lcms2
multi-tty make-network-process native-compile emacs)

Memory information:
((conses 16 84977 6469)
 (symbols 48 7541 0)
 (strings 32 21297 2294)
 (string-bytes 1 660223)
 (vectors 16 16575)
 (vector-slots 8 342018 13903)
 (floats 8 72 31)
 (intervals 56 577 0)
 (buffers 984 14))

-- 
Eason Huang

bug#58928: 29.0.50; overlays in org-mode are disrupted after call `org-capture`

[Gerd Möllmann]

Eason Huang <aqua0210@foxmail.com> writes:

> Stefan Monnier <monnier@iro.umontreal.ca> writes:
>
>>> I don't know, but my guess is that the indirect buffer thing is already
>>> a good starting point for Stefan/Matt.
>>
>> Indeed.  I believe it is now fixed.  Please confirm,
>
> Hi stefan,
>
> I tried the latest commit, but thhis issue is still existed on macOS and
> Windows 10.
>
> I tested commit 8a5678906fa1b899c4d111e5ee4334b278f50d48 on Windows 10,
> and commit 2eb2eb0c9ead205935be58ce9bb54465d48b20c8 on macOS Ventura
> (13.0)

I can't reproduce it anymore (macOS 13, but that shouldn't matter).  I'm
on d16494cffbed79a916482558ae5ed1bdcc67c88d.

Did you anything different from what you did before?

bug#58928: 29.0.50; overlays in org-mode are disrupted after call `org-capture`

[Eason Huang]

[-- Attachment #1: Type: text/plain, Size: 1627 bytes --]

Hi Gerd,

> 2022年11月3日 22:25，Gerd Möllmann <gerd.moellmann@gmail.com>writes：
> 
> Eason Huang <aqua0210@foxmail.com> writes:
> 
>> Stefan Monnier <monnier@iro.umontreal.ca> writes:
>> 
>>>> I don't know, but my guess is that the indirect buffer thing is already
>>>> a good starting point for Stefan/Matt.
>>> 
>>> Indeed.  I believe it is now fixed.  Please confirm,
>> 
>> Hi stefan,
>> 
>> I tried the latest commit, but thhis issue is still existed on macOS and
>> Windows 10.
>> 
>> I tested commit 8a5678906fa1b899c4d111e5ee4334b278f50d48 on Windows 10,
>> and commit 2eb2eb0c9ead205935be58ce9bb54465d48b20c8 on macOS Ventura
>> (13.0)
> 
> I can't reproduce it anymore (macOS 13, but that shouldn't matter).  I'm
> on d16494cffbed79a916482558ae5ed1bdcc67c88d.
> 
> Did you anything different from what you did before?


No, I did the same as before.

I can reproduce the following steps, may be you can have try:

1. emacs -Q , launch Emacs
2. M-x org-capture, and then type `t` (task)
   Now the buffer CAPTURE-.notes will open with contents as bellow:

   ** TODO (now the cursor is here, you can type some characters, such as "test1", and then type C-C C-c )
  [2022-11-03 Thu]

3. C-x b, switch-to-buffer .notes
  The contents is as below:
* Tasks
** TODO test1(move cusror here and type TAB)
  [2022-11-03 Thu]

4. Now the .notes buffer is like this:
* Tasks
** TODO test1...
5. Try step 2 again, now you will see the issue. Now the .notes buffer looks like this:
* Tasks
** TODO test1...TODO
  [2022-11-03 Thu]
  test1



------
Eason Huang



[-- Attachment #2: Type: text/html, Size: 2548 bytes --]

bug#58928: 29.0.50; overlays in org-mode are disrupted after call `org-capture`

[Eason Huang]

Gerd Möllmann <gerd.moellmann@gmail.com> writes:

> Eason Huang <aqua0210@foxmail.com> writes:
>
>> Stefan Monnier <monnier@iro.umontreal.ca> writes:
>>
>>>> I don't know, but my guess is that the indirect buffer thing is already
>>>> a good starting point for Stefan/Matt.
>>>
>>> Indeed.  I believe it is now fixed.  Please confirm,
>>
>> Hi stefan,
>>
>> I tried the latest commit, but thhis issue is still existed on macOS and
>> Windows 10.
>>
>> I tested commit 8a5678906fa1b899c4d111e5ee4334b278f50d48 on Windows 10,
>> and commit 2eb2eb0c9ead205935be58ce9bb54465d48b20c8 on macOS Ventura
>> (13.0)
>
> I can't reproduce it anymore (macOS 13, but that shouldn't matter).  I'm
> on d16494cffbed79a916482558ae5ed1bdcc67c88d.
>
> Did you anything different from what you did before?

No, I did the same as before.

I can reproduce the following steps, may be you can have try:

1. emacs -Q , launch Emacs
2. M-x org-capture, and then type `t` (task)
   Now the buffer CAPTURE-.notes will open with contents as bellow:

   ** TODO (now the cursor is here, you can type some characters, such as "test1", and then type C-C C-c )
  [2022-11-03 Thu]

3. C-x b, switch-to-buffer .notes
  The contents is as below:
* Tasks
** TODO test1(move cusror here and type TAB)
  [2022-11-03 Thu]

4. Now the .notes buffer is like this:
* Tasks
** TODO test1...
5. Try step 2 again, now you will see the issue. Now the .notes buffer looks like this:
* Tasks
** TODO test1...TODO
  [2022-11-03 Thu]
  test1


I sented by mac Mail, but look like it failed, so I sent it again with
Gnus. Sorry for sending two time.

-- 
Eason Huang

bug#58928: 29.0.50; overlays in org-mode are disrupted after call `org-capture`

[Gerd Möllmann]

On 03.11.22 15:57, Eason Huang wrote:
> I can reproduce the following steps, may be you can have try:
> 
> 1. emacs -Q , launch Emacs
> 2. M-x org-capture, and then type `t` (task)
>     Now the buffer CAPTURE-.notes will open with contents as bellow:
> 
>     ** TODO (now the cursor is here, you can type some characters, such 
> as "test1", and then type C-C C-c )
>    [2022-11-03 Thu]
> 
> 3. C-x b, switch-to-buffer .notes
>    The contents is as below:
> * Tasks
> ** TODO test1(move cusror here and type TAB)
>    [2022-11-03 Thu]
> 
> 4. Now the .notes buffer is like this:
> * Tasks
> ** TODO test1...
> 5. Try step 2 again, now you will see the issue. Now the .notes buffer 
> looks like this:
> * Tasks
> ** TODO test1...TODO
>    [2022-11-03 Thu]
>    test1

I do indeed see something like that, except that it is
"** TODO test1...** TODO", i.e. the second TODO has the stars in front
of it.

Typing in the CAPTURE buffer shows the typed text in .notes after the
second TODO, and the ellipsis doesn't move.

I guess the second TODO should be/was on a new line?

bug#58928: 29.0.50; overlays in org-mode are disrupted after call `org-capture`

[Eason Huang]

Hi Gerd,

> 
> I do indeed see something like that, except that it is
> "** TODO test1...** TODO", i.e. the second TODO has the stars in front
> of it.

I have same result as you. It was my typo for missing stars before second TODO

> Typing in the CAPTURE buffer shows the typed text in .notes after the
> second TODO, and the ellipsis doesn't move.
> 
> I guess the second TODO should be/was on a new line?

I agree. The second TODO should be on a new line.



Eason Huang
aqua0210@foxmail.com

bug#58928: 29.0.50; overlays in org-mode are disrupted after call `org-capture`

[Eason Huang]

Hi Gerd,

> Gerd Möllmann <gerd.moellmann@gmail.com> writes:
>
>
> I do indeed see something like that, except that it is
> "** TODO test1...** TODO", i.e. the second TODO has the stars in front
> of it.

I have same result as you. It was my typo for missing stars before
second TODO in the last email. Sorry!

> Typing in the CAPTURE buffer shows the typed text in .notes after the
> second TODO, and the ellipsis doesn't move.

> I guess the second TODO should be/was on a new line?

I agree. The second TODO should be on a new line.

-- 
Eason Huang

bug#58928: 29.0.50; overlays in org-mode are disrupted after call `org-capture`

[Gerd Möllmann]

On 03.11.22 16:45, Eason Huang wrote:
> 
> Hi Gerd,
> 
>> Gerd Möllmann <gerd.moellmann@gmail.com> writes:
>>
>>
>> I do indeed see something like that, except that it is
>> "** TODO test1...** TODO", i.e. the second TODO has the stars in front
>> of it.
> 
> I have same result as you. It was my typo for missing stars before
> second TODO in the last email. Sorry!
> 
>> Typing in the CAPTURE buffer shows the typed text in .notes after the
>> second TODO, and the ellipsis doesn't move.
> 
>> I guess the second TODO should be/was on a new line?
> 
> I agree. The second TODO should be on a new line.

Then it's probably something we didn't have yet.  Maybe a off-by-one 
error somewhere.

bug#58928: 29.0.50; overlays in org-mode are disrupted after call `org-capture`

[Matt Armstrong]

Eason Huang <aqua0210@foxmail.com> writes:

> I can reproduce the following steps, may be you can have try:
>
> 1. emacs -Q , launch Emacs
> 2. M-x org-capture, and then type `t` (task)
>    Now the buffer CAPTURE-.notes will open with contents as bellow:
>
>    ** TODO (now the cursor is here, you can type some characters, such as "test1", and then type C-C C-c )
>   [2022-11-03 Thu]
>
> 3. C-x b, switch-to-buffer .notes
>   The contents is as below:
> * Tasks
> ** TODO test1(move cusror here and type TAB)
>   [2022-11-03 Thu]
>
> 4. Now the .notes buffer is like this:
> * Tasks
> ** TODO test1...
> 5. Try step 2 again, now you will see the issue. Now the .notes buffer looks like this:
> * Tasks
> ** TODO test1...TODO
>   [2022-11-03 Thu]
>   test1

I can reproduce this too.  Note that this is a new set of repro steps
that are easier to do after "emacs -Q".  Thanks Eason!

I think it is a different bug now visible after Stefan fixed the first
one.

It looks like `insert-before-markers' behaves differently for overlays
ending at `point'.  Indirect buffers are not required to reproduce it.
These tests now fail when they used to pass:

(ert-deftest test-overlay-insert-before-markers-empty ()
  (with-temp-buffer
    (insert "1234")
    (let ((overlay (make-overlay 2 2)))
      (goto-char 2)
      (insert-before-markers "x")
      (should (equal 3 (overlay-end overlay))))))

(ert-deftest test-overlay-insert-before-markers-non-empty ()
  (with-temp-buffer
    (insert "1234")
    (let ((overlay (make-overlay 2 3)))
      (goto-char 3)
      (insert-before-markers "x")
      (should (equal 4 (overlay-end overlay))))))

I am not sure that fixing this behavior difference will fix Eason's
repro steps, but it is worth fixing the above anyway.

Stefan, I won't have time to work on this until tomorrow.  If you have
the time I won't mind if you take a crack at a fix as you'll be about
100 times faster.  ;-)

bug#58928: 29.0.50; overlays in org-mode are disrupted after call `org-capture`

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

> It looks like `insert-before-markers' behaves differently for overlays
> ending at `point'.  Indirect buffers are not required to reproduce it.
> These tests now fail when they used to pass:

I pushed a fix for this problem, but it doesn't solve Eason's case (it
actually makes it worse: instead of the LF becoming invisible, it's the
whole second entry which becomes swallowed/covered by the fold of the first).


        Stefan

bug#58928: 29.0.50; overlays in org-mode are disrupted after call `org-capture`

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

> 4. Now the .notes buffer is like this:
> * Tasks
> ** TODO test1...
> 5. Try step 2 again, now you will see the issue. Now the .notes buffer looks like this:
> * Tasks
> ** TODO test1...TODO
>   [2022-11-03 Thu]
>   test1

With the latest changes I pushed, at the end of step 5 I see only:

    ** TODO test1...

This doesn't look completely right (the "..." hides both the "test1"
todo and the subsequent one), but I see exactly the same when
I try it on Emacs-28, so are you sure it's something that changed with
the new overlays code?

This behavior is different from the behavior I see in Emacs-27, where
I get:

    ** TODO test1...
    ** TODO test2
      [2022-11-03 Thu]

but the difference comes from where the invisible overlay is placed:
when the `.notes` buffer just contains:

    * Tasks
    ** TODO test1
      [2022-11-03 Thu]

if you hit TAB with cursor on the TODO line, in both Emacs-28/29 and
Emacs-27 I see:

    * Tasks
    ** TODO test1...

but if I `M->` to go to the end of buffer, I see that in Emacs-27, the
final LF character is not invisible (i.e. my cursor is displayed below
the "**") whereas with Emacs-28/29 my cursor ends up right after the
"...".


        Stefan

bug#58928: 29.0.50; overlays in org-mode are disrupted after call `org-capture`

[Matt Armstrong]

[-- Attachment #1: Type: text/plain, Size: 1954 bytes --]

Matt Armstrong <matt@rfc20.org> writes:

> Eason Huang <aqua0210@foxmail.com> writes:
>
>> I can reproduce the following steps, may be you can have try:
>>
>> 1. emacs -Q , launch Emacs
>> 2. M-x org-capture, and then type `t` (task)
>>    Now the buffer CAPTURE-.notes will open with contents as bellow:
>>
>>    ** TODO (now the cursor is here, you can type some characters, such as "test1", and then type C-C C-c )
>>   [2022-11-03 Thu]
>>
>> 3. C-x b, switch-to-buffer .notes
>>   The contents is as below:
>> * Tasks
>> ** TODO test1(move cusror here and type TAB)
>>   [2022-11-03 Thu]
>>
>> 4. Now the .notes buffer is like this:
>> * Tasks
>> ** TODO test1...
>> 5. Try step 2 again, now you will see the issue. Now the .notes buffer looks like this:
>> * Tasks
>> ** TODO test1...TODO
>>   [2022-11-03 Thu]
>>   test1
>
> I can reproduce this too.  Note that this is a new set of repro steps
> that are easier to do after "emacs -Q".  Thanks Eason!

See attached fix to fix the bug found by Eason's steps quoted above.
The bug was that we ignored the "before markers" part of
`insert-before-markers'.

Stefan, this and a couple other minor patches are in
https://git.sr.ht/~matta/emacs/log/scratch/matta/for_stefan

Now, with respect to Eason's repro above, Emacs is "bug/behavior
compatible" with itself before the noverlay merge to master.

There may still be an org-mode bug here.  In step 3 above org puts an
invisible overlay with "ellipsis" over the "TODO test1" heading and
content.  When it later appends the new heading via org-capture it does
so with `insert-before-markers' which causes the end of the "ellipsis"
overlay to cover the new entry too, which later confuses other org
functions until the heading is expanded and the overlay is deleted.
This seems wrong and doesn't happen in Emacs 27.  But, it does happen as
of commit 69121c33e4a11805bf6438131c8aec72411a0e5d (the predecessor to
Stefan's noverlay->master merge commit).


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0003-Fix-insert-before-markers-for-the-new-overlays-imple.patch --]
[-- Type: text/x-diff, Size: 12203 bytes --]

From 23715529c7b87dfbed7810f674a161ddb4fbd909 Mon Sep 17 00:00:00 2001
From: Matt Armstrong <matt@rfc20.org>
Date: Thu, 3 Nov 2022 20:57:29 -0700
Subject: [PATCH 3/3] Fix insert-before-markers for the new overlays
 implementation.

* src/itree.h: Add a before_markers arg to itree_insert_gap.
* src/itree.c (itree_insert_gap): Use it, fixing bug#58928.
* src/lisp.h: Add a before_markers arg to adjust_overlays_for_insert.
* src/buffer.c (adjust_overlays_for_insert): Pass it through to
itree_insert_gap.
* src/insdel.c (insert_1_both): Pass before_markers through to
adjust_overlays_for_insert.
(insert_from_string_1): Ditto.
(insert_from_gap): Pass before_markers=false to
adjust_overlays_for_insert.
(insert_from_buffer_1): ditto.
(adjust_after_replace): ditto.
(replace_range): ditto.
(replace_range_2): ditto.
* test/src/buffer-tests.el (buffer-tests-overlay-string): New helper.
(test-overlay-insert-before-markers-at-start): New test.
(test-overlay-insert-before-markers-at-end): New test.
---
 src/buffer.c             | 11 ++++---
 src/insdel.c             | 14 ++++-----
 src/itree.c              | 28 +++++++++++-------
 src/itree.h              |  3 +-
 src/lisp.h               |  2 +-
 test/src/buffer-tests.el | 62 ++++++++++++++++++++++++++++++++++++++++
 6 files changed, 97 insertions(+), 23 deletions(-)

diff --git a/src/buffer.c b/src/buffer.c
index 3129aa2890e..5e15e9e1ec3 100644
--- a/src/buffer.c
+++ b/src/buffer.c
@@ -3454,20 +3454,23 @@ overlay_strings (ptrdiff_t pos, struct window *w, unsigned char **pstr)
 
 \f
 void
-adjust_overlays_for_insert (ptrdiff_t pos, ptrdiff_t length)
+adjust_overlays_for_insert (ptrdiff_t pos, ptrdiff_t length,
+			    bool before_markers)
 {
   if (!current_buffer->indirections)
-    itree_insert_gap (current_buffer->overlays, pos, length);
+    itree_insert_gap (current_buffer->overlays, pos, length,
+		      before_markers);
   else
     {
       struct buffer *base = current_buffer->base_buffer
                             ? current_buffer->base_buffer
                             : current_buffer;
       Lisp_Object tail, other;
-      itree_insert_gap (base->overlays, pos, length);
+      itree_insert_gap (base->overlays, pos, length, before_markers);
       FOR_EACH_LIVE_BUFFER (tail, other)
         if (XBUFFER (other)->base_buffer == base)
-          itree_insert_gap (XBUFFER (other)->overlays, pos, length);
+	  itree_insert_gap (XBUFFER (other)->overlays, pos, length,
+			    before_markers);
     }
 }
 
diff --git a/src/insdel.c b/src/insdel.c
index 6d56a76c77a..33958f53e9c 100644
--- a/src/insdel.c
+++ b/src/insdel.c
@@ -917,7 +917,7 @@ insert_1_both (const char *string,
   if (Z - GPT < END_UNCHANGED)
     END_UNCHANGED = Z - GPT;
 
-  adjust_overlays_for_insert (PT, nchars);
+  adjust_overlays_for_insert (PT, nchars, before_markers);
   adjust_markers_for_insert (PT, PT_BYTE,
 			     PT + nchars, PT_BYTE + nbytes,
 			     before_markers);
@@ -1043,7 +1043,7 @@ insert_from_string_1 (Lisp_Object string, ptrdiff_t pos, ptrdiff_t pos_byte,
   if (Z - GPT < END_UNCHANGED)
     END_UNCHANGED = Z - GPT;
 
-  adjust_overlays_for_insert (PT, nchars);
+  adjust_overlays_for_insert (PT, nchars, before_markers);
   adjust_markers_for_insert (PT, PT_BYTE, PT + nchars,
 			     PT_BYTE + outgoing_nbytes,
 			     before_markers);
@@ -1115,7 +1115,7 @@ insert_from_gap (ptrdiff_t nchars, ptrdiff_t nbytes, bool text_at_gap_tail)
 
   insert_from_gap_1 (nchars, nbytes, text_at_gap_tail);
 
-  adjust_overlays_for_insert (ins_charpos, nchars);
+  adjust_overlays_for_insert (ins_charpos, nchars, false);
   adjust_markers_for_insert (ins_charpos, ins_bytepos,
 			     ins_charpos + nchars, ins_bytepos + nbytes, 0);
 
@@ -1257,7 +1257,7 @@ insert_from_buffer_1 (struct buffer *buf,
   if (Z - GPT < END_UNCHANGED)
     END_UNCHANGED = Z - GPT;
 
-  adjust_overlays_for_insert (PT, nchars);
+  adjust_overlays_for_insert (PT, nchars, false);
   adjust_markers_for_insert (PT, PT_BYTE, PT + nchars,
 			     PT_BYTE + outgoing_nbytes,
 			     0);
@@ -1323,7 +1323,7 @@ adjust_after_replace (ptrdiff_t from, ptrdiff_t from_byte,
   record_insert (from, len);
 
   if (len > nchars_del)
-    adjust_overlays_for_insert (from, len - nchars_del);
+    adjust_overlays_for_insert (from, len - nchars_del, false);
   else if (len < nchars_del)
     adjust_overlays_for_delete (from, nchars_del - len);
 
@@ -1513,7 +1513,7 @@ replace_range (ptrdiff_t from, ptrdiff_t to, Lisp_Object new,
   /* Adjust the overlay center as needed.  This must be done after
      adjusting the markers that bound the overlays.  */
   adjust_overlays_for_delete (from, nchars_del);
-  adjust_overlays_for_insert (from, inschars);
+  adjust_overlays_for_insert (from, inschars, false);
 
   offset_intervals (current_buffer, from, inschars - nchars_del);
 
@@ -1648,7 +1648,7 @@ replace_range_2 (ptrdiff_t from, ptrdiff_t from_byte,
      adjusting the markers that bound the overlays.  */
   if (nchars_del != inschars)
     {
-      adjust_overlays_for_insert (from, inschars);
+      adjust_overlays_for_insert (from, inschars, false);
       adjust_overlays_for_delete (from + inschars, nchars_del);
     }
 
diff --git a/src/itree.c b/src/itree.c
index bd4e8cc5740..0b56feed43f 100644
--- a/src/itree.c
+++ b/src/itree.c
@@ -1190,11 +1190,14 @@ itree_iterator_finish (struct itree_iterator *iter)
 
 /* Insert a gap at POS of length LENGTH expanding all intervals
    intersecting it, while respecting their rear_advance and
-   front_advance setting. */
+   front_advance setting.
+
+   When BEFORE_MARKERS, all overlays beginning/ending at POS are
+   treated as if their front_advance/rear_advance was true. */
 
 void
-itree_insert_gap (struct itree_tree *tree,
-		  ptrdiff_t pos, ptrdiff_t length)
+itree_insert_gap (struct itree_tree *tree, ptrdiff_t pos,
+		  ptrdiff_t length, bool before_markers)
 {
   if (!tree || length <= 0 || tree->root == NULL)
     return;
@@ -1202,14 +1205,16 @@ itree_insert_gap (struct itree_tree *tree,
 
   /* FIXME: Don't allocate iterator/stack anew every time. */
 
-  /* Nodes with front_advance starting at pos may mess up the tree
-     order, so we need to remove them first. */
+  /* Nodes starting at pos with front_advance (or before_markers) may
+     mess up the tree order, so we need to remove them first. */
   struct interval_stack *saved = interval_stack_create (0);
   struct itree_node *node = NULL;
   ITREE_FOREACH (node, tree, pos, pos + 1, PRE_ORDER)
     {
-      if (node->begin == pos && node->front_advance
-	  && (node->begin != node->end || node->rear_advance))
+      if (node->begin == pos
+	  && (node->front_advance || before_markers)
+	  && (node->begin != node->end || node->rear_advance
+	      || before_markers))
 	interval_stack_push (saved, node);
     }
   for (int i = 0; i < saved->length; ++i)
@@ -1246,7 +1251,9 @@ itree_insert_gap (struct itree_tree *tree,
 	  /* node->begin == pos implies no front-advance. */
 	  if (node->begin > pos)
 	    node->begin += length;
-	  if (node->end > pos || (node->end == pos && node->rear_advance))
+	  if (node->end > pos
+	      || (node->end == pos
+		  && (node->rear_advance || before_markers)))
 	    {
 	      node->end += length;
 	      eassert (node != NULL);
@@ -1256,7 +1263,8 @@ itree_insert_gap (struct itree_tree *tree,
       interval_stack_destroy (stack);
     }
 
-  /* Reinsert nodes starting at POS having front-advance. */
+  /* Reinsert nodes starting at POS with front-advance (or
+     before_markers). */
   uintmax_t notick = tree->otick;
   nodeptr_and_flag nav;
   while ((nav = interval_stack_pop (saved),
@@ -1264,7 +1272,7 @@ itree_insert_gap (struct itree_tree *tree,
     {
       eassert (node->otick == ootick);
       node->begin += length;
-      if (node->end != pos || node->rear_advance)
+      if (node->end != pos || (node->rear_advance || before_markers))
 	node->end += length;
       node->otick = notick;
       interval_tree_insert (tree, node);
diff --git a/src/itree.h b/src/itree.h
index c6b68d36672..3ef9e76812a 100644
--- a/src/itree.h
+++ b/src/itree.h
@@ -119,7 +119,8 @@ #define ITREE_H
 			  ptrdiff_t, ptrdiff_t);
 extern struct itree_node *itree_remove (struct itree_tree *,
 					struct itree_node *);
-extern void itree_insert_gap (struct itree_tree *, ptrdiff_t, ptrdiff_t);
+extern void itree_insert_gap (struct itree_tree *, ptrdiff_t,
+			      ptrdiff_t, bool);
 extern void itree_delete_gap (struct itree_tree *, ptrdiff_t, ptrdiff_t);
 
 /* Iteration functions.  Almost all code should use ITREE_FOREACH
diff --git a/src/lisp.h b/src/lisp.h
index d87f9549382..472472d87f8 100644
--- a/src/lisp.h
+++ b/src/lisp.h
@@ -4690,7 +4690,7 @@ XMODULE_FUNCTION (Lisp_Object o)
 extern bool mouse_face_overlay_overlaps (Lisp_Object);
 extern Lisp_Object disable_line_numbers_overlay_at_eob (void);
 extern AVOID nsberror (Lisp_Object);
-extern void adjust_overlays_for_insert (ptrdiff_t, ptrdiff_t);
+extern void adjust_overlays_for_insert (ptrdiff_t, ptrdiff_t, bool);
 extern void adjust_overlays_for_delete (ptrdiff_t, ptrdiff_t);
 extern void fix_start_end_in_overlays (ptrdiff_t, ptrdiff_t);
 extern void report_overlay_modification (Lisp_Object, Lisp_Object, bool,
diff --git a/test/src/buffer-tests.el b/test/src/buffer-tests.el
index a4a1f609fd2..c8bd86ca9f1 100644
--- a/test/src/buffer-tests.el
+++ b/test/src/buffer-tests.el
@@ -1165,6 +1165,68 @@ test-delete-all-overlay-1
     (should-not (delete-all-overlays))))
 
 
+;; +==========================================================================+
+;; | insert-before-markers and overlays
+;; +==========================================================================+
+
+(defun buffer-tests-overlay-string (overlay)
+  (buffer-substring-no-properties
+   (overlay-start overlay)
+   (overlay-end overlay)))
+
+(ert-deftest test-overlay-insert-before-markers-at-start ()
+  "`insert-before-markers' always advances an overlay's start.
+Test both front-advance and non-front-advance overlays."
+  (with-temp-buffer
+    (insert "1234")
+    (let ((front-advance (make-overlay 2 3 nil t))
+          (default-advance (make-overlay 2 3 nil nil)))
+      (goto-char 2)
+      (insert-before-markers "x")
+      (should (equal "2" (buffer-tests-overlay-string front-advance)))
+      (should (equal "2" (buffer-tests-overlay-string default-advance))))))
+
+(ert-deftest test-overlay-insert-before-markers-at-end ()
+  "`insert-before-markers' always advances an overlay's start.
+Test both front-advance and non-front-advance overlays."
+  (with-temp-buffer
+    (insert "1234")
+    (let ((rear-advance (make-overlay 2 3 nil nil t))
+          (default-advance (make-overlay 2 3 nil nil nil)))
+      (goto-char 3)
+      (insert-before-markers "x")
+      (should (equal "2x" (buffer-tests-overlay-string rear-advance)))
+      (should (equal "2x" (buffer-tests-overlay-string default-advance))))))
+
+;; (ert-deftest test-overlay-insert-before-markers-at-start ()
+;;   (with-temp-buffer
+;;     (insert "1234")
+;;     (let ((front-advance (make-overlay 2 3 nil 'front-advance))
+;;           (default-advance (make-overlay 2 3 nil nil)))
+;;       (goto-char 2)
+;;       (insert-before-markers "x")
+;;       (should (equal -1 (overlay-start front-advance)))
+;;       (should (equal -1 (overlay-start default-advance)))
+;;       (should (equal -1 (overlay-end overlay)))
+;;       (should (equal -1 (overlay-end overlay)))
+;;       )))
+
+;; (ert-deftest test-overlay-insert-before-markers-empty ()
+;;   (with-temp-buffer
+;;     (insert "1234")
+;;     (let ((overlay (make-overlay 2 2)))
+;;       (goto-char 2)
+;;       (insert-before-markers "x")
+;;       (should (equal 3 (overlay-end overlay))))))
+
+;; (ert-deftest test-overlay-insert-before-markers-non-empty ()
+;;   (with-temp-buffer
+;;     (insert "1234")
+;;     (let ((overlay (make-overlay 2 3)))
+;;       (goto-char 3)
+;;       (insert-before-markers "x")
+;;       (should (equal 4 (overlay-end overlay))))))
+
 ;; +==========================================================================+
 ;; | get-pos-property
 ;; +==========================================================================+
-- 
2.35.1

bug#58928: 29.0.50; overlays in org-mode are disrupted after call `org-capture`

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

> See attached fix to fix the bug found by Eason's steps quoted above.
> The bug was that we ignored the "before markers" part of
> `insert-before-markers'.

I pushed a similar fix a few hours earlier, sorry :-)

> Stefan, this and a couple other minor patches are in
> https://git.sr.ht/~matta/emacs/log/scratch/matta/for_stefan

Could you rebase the remaining minor patches?


        Stefan

bug#58928: 29.0.50; overlays in org-mode are disrupted after call `org-capture`

[Matt Armstrong]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> 4. Now the .notes buffer is like this:
>> * Tasks
>> ** TODO test1...
>> 5. Try step 2 again, now you will see the issue. Now the .notes buffer looks like this:
>> * Tasks
>> ** TODO test1...TODO
>>   [2022-11-03 Thu]
>>   test1
>
> With the latest changes I pushed, at the end of step 5 I see only:
>
>     ** TODO test1...
>
> This doesn't look completely right (the "..." hides both the "test1"
> todo and the subsequent one), but I see exactly the same when
> I try it on Emacs-28, so are you sure it's something that changed with
> the new overlays code?

It isn't a regression in Emacs' overlay implementation.  It is a bug
introduced in the org-mode shipped with Emacs 28.  I verified that this
is fixed in current org mode dev version.

So, I think we're done here!

bug#58928: 29.0.50; overlays in org-mode are disrupted after call `org-capture`

[Matt Armstrong]

[-- Attachment #1: Type: text/plain, Size: 1027 bytes --]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> See attached fix to fix the bug found by Eason's steps quoted above.
>> The bug was that we ignored the "before markers" part of
>> `insert-before-markers'.
>
> I pushed a similar fix a few hours earlier, sorry :-)

Your fix was better!  ;)


>> Stefan, this and a couple other minor patches are in
>> https://git.sr.ht/~matta/emacs/log/scratch/matta/for_stefan
>
> Could you rebase the remaining minor patches?

Done (patches here and at my sr.ht).  Basically, I rewrote the test in a
way that was clearer, at lest for me, and more exhaustive.  Moved them
down to the other tests related to inserting and moving overlays.  Used
explicit buffer positions instead of (point) as a relative reference
(which confused me and seemed unrelated, since neither
`insert-before-markers` nor the overlay code is ever concerned with
(point).  If you don't like that, we I think we should still tweak the
one new test to not attempt (goto-char (2+ (point))) when point is at
eob.  :-)


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-Tweak-the-overlay-related-insert-before-markers-test.patch --]
[-- Type: text/x-diff, Size: 4378 bytes --]

From 4e66e39cdcd27d04f21d2459516d36ea22727bca Mon Sep 17 00:00:00 2001
From: Matt Armstrong <matt@rfc20.org>
Date: Fri, 4 Nov 2022 15:02:17 -0700
Subject: [PATCH 1/2] Tweak the overlay related `insert-before-markers' tests

* test/src/buffer-tests.el (test-overlay-insert-before-markers-empty):
Move code down to the other tests related to insertion.  Test all
front/rear insert combinations.  To make the test more clear, at least
to me, hard code all character positions.
(test-overlay-insert-before-markers-at-start): For both front-advance
modes verify that `insert-before-markers' at and overlay's start
advances it.
(test-overlay-insert-before-markers-at-end): For both rear-advance
modes test that `insert-before-markers' at an overlay's end advances
it.
(test-overlay-insert-before-markers-non-empty): Delete, replaced by
the two tests above.
---
 test/src/buffer-tests.el | 61 +++++++++++++++++++++++++---------------
 1 file changed, 39 insertions(+), 22 deletions(-)

diff --git a/test/src/buffer-tests.el b/test/src/buffer-tests.el
index a39d7d51de1..0e9e84ef7a1 100644
--- a/test/src/buffer-tests.el
+++ b/test/src/buffer-tests.el
@@ -528,28 +528,6 @@ K
 (deftest-overlay-start/end-1 L (1 0) (1 1))
 (deftest-overlay-start/end-1 M (0 0) (1 1))
 
-(ert-deftest test-overlay-insert-before-markers-empty ()
-  (with-temp-buffer
-    (insert "1234")
-    (goto-char (1+ (point-min)))
-    (let ((overlay (make-overlay (point) (point))))
-      (insert-before-markers "x")
-      (should (equal (point) (overlay-end overlay)))
-      (should (equal (point) (overlay-start overlay))))))
-
-(ert-deftest test-overlay-insert-before-markers-non-empty ()
-  (with-temp-buffer
-    (insert "1234")
-    (goto-char (+ 2 (point)))
-    (let ((overlay (make-overlay (1- (point)) (point))))
-      (insert-before-markers "x")
-      (should (equal (point) (overlay-end overlay)))
-      (should (equal (- (point) 2) (overlay-start overlay)))
-      (forward-char -2)
-      (insert-before-markers "y")
-      (should (equal (+ 2 (point)) (overlay-end overlay)))
-      (should (equal (point) (overlay-start overlay))))))
-
 (ert-deftest test-overlay-start/end-2 ()
   (should-not (overlay-start (with-temp-buffer (make-overlay 1 1))))
   (should-not (overlay-end (with-temp-buffer (make-overlay 1 1)))))
@@ -1315,7 +1293,46 @@ test-moving-insert-2
       (delete-overlay left)
       (should (= 2 (length (overlays-in 1 (point-max))))))))
 
+;; +==========================================================================+
+;; | Moving overlays with insert-before-markers
+;; +==========================================================================+
 
+(ert-deftest test-overlay-insert-before-markers-at-start ()
+  "`insert-before-markers' always advances an overlay's start.
+Test both front-advance and non-front-advance overlays."
+  (dolist (front-advance '(nil t))
+    (ert-info ((format "front-advance %S" front-advance))
+      (with-temp-buffer
+        (insert "1234")
+        (let ((overlay (make-overlay 2 3 nil front-advance nil)))
+          (goto-char 2)
+          (insert-before-markers "x")
+          (should (equal 3 (overlay-start overlay)))
+          (should (equal 4 (overlay-end overlay))))))))
+
+(ert-deftest test-overlay-insert-before-markers-at-end ()
+  "`insert-before-markers' always advances an overlay's end.
+Test both rear-advance and non-rear-advance overlays."
+  (dolist (rear-advance '(nil t))
+    (ert-info ((format "rear-advance %S" rear-advance))
+      (with-temp-buffer
+        (insert "1234")
+        (let ((overlay (make-overlay 2 3 nil nil rear-advance)))
+          (goto-char 3)
+          (insert-before-markers "x")
+          (should (equal 2 (overlay-start overlay)))
+          (should (equal 4 (overlay-end overlay))))))))
+
+(ert-deftest test-overlay-insert-before-markers-empty ()
+  (dolist (advance-args '((nil nil) (t nil) (nil t) (t t)))
+    (ert-info ((format "advance args %S" advance-args))
+      (with-temp-buffer
+        (insert "1234")
+        (let ((overlay (apply #'make-overlay 2 2 nil advance-args)))
+          (goto-char 2)
+          (insert-before-markers "x")
+          (should (equal 3 (overlay-start overlay)))
+          (should (equal 3 (overlay-end overlay))))))))
 
 ;; +==========================================================================+
 ;; | Moving by deletions
-- 
2.35.1


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #3: 0002-Minor-tweaks-to-the-fix-for-insert-before-markers-ov.patch --]
[-- Type: text/x-diff, Size: 1598 bytes --]

From e7e66c30430b4e331a9285159851492fd4c78c3c Mon Sep 17 00:00:00 2001
From: Matt Armstrong <matt@rfc20.org>
Date: Fri, 4 Nov 2022 15:24:40 -0700
Subject: [PATCH 2/2] Minor tweaks to the fix for `insert-before-markers'
 overlay fix

(bug#58928)

* src/buffer.c (adjust_overlays_for_insert): wrap to less than 80
chars.
* src/itree.c: document BEFORE_MARKERS.
---
 src/buffer.c | 3 ++-
 src/itree.c  | 5 ++++-
 2 files changed, 6 insertions(+), 2 deletions(-)

diff --git a/src/buffer.c b/src/buffer.c
index 745e62f53f7..390ccff5c8a 100644
--- a/src/buffer.c
+++ b/src/buffer.c
@@ -3467,7 +3467,8 @@ adjust_overlays_for_insert (ptrdiff_t pos, ptrdiff_t length, bool before_markers
       itree_insert_gap (base->overlays, pos, length, before_markers);
       FOR_EACH_LIVE_BUFFER (tail, other)
         if (XBUFFER (other)->base_buffer == base)
-          itree_insert_gap (XBUFFER (other)->overlays, pos, length, before_markers);
+	  itree_insert_gap (XBUFFER (other)->overlays, pos, length,
+			    before_markers);
     }
 }
 
diff --git a/src/itree.c b/src/itree.c
index cd37da18b89..b744b8423a2 100644
--- a/src/itree.c
+++ b/src/itree.c
@@ -1183,7 +1183,10 @@ itree_iterator_finish (struct itree_iterator *iter)
 
 /* Insert a gap at POS of length LENGTH expanding all intervals
    intersecting it, while respecting their rear_advance and
-   front_advance setting. */
+   front_advance setting.
+
+   When BEFORE_MARKERS, all overlays beginning/ending at POS are
+   treated as if their front_advance/rear_advance was true. */
 
 void
 itree_insert_gap (struct itree_tree *tree,
-- 
2.35.1

bug#58928: 29.0.50; overlays in org-mode are disrupted after call `org-capture`

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

> Used explicit buffer positions instead of (point) as a relative
> reference

I won't go changing all the tests in `src/buffer-tests.el`, but I consider
it a bug to assume that (point-min) is 1.


        Stefan

bug#58928: 29.0.50; overlays in org-mode are disrupted after call `org-capture`

[Ihor Radchenko]

Matt Armstrong <matt@rfc20.org> writes:

> It isn't a regression in Emacs' overlay implementation.  It is a bug
> introduced in the org-mode shipped with Emacs 28.  I verified that this
> is fixed in current org mode dev version.

Note that current dev version of Org does not use overlays for folding.
If you want to test the behaviour with overlays, please set
org-fold-core-style to 'overlays before loading Org.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

bug#58928: 29.0.50; overlays in org-mode are disrupted after call `org-capture`

[Eli Zaretskii]

> From: Matt Armstrong <matt@rfc20.org>
> Cc: Eason Huang <aqua0210@foxmail.com>, Gerd Möllmann
>  <gerd.moellmann@gmail.com>, Ihor Radchenko <yantar92@posteo.net>,
>  58928@debbugs.gnu.org, Eli Zaretskii <eliz@gnu.org>, Eli Qian
>  <eli.q.qian@gmail.com>
> Date: Fri, 04 Nov 2022 15:47:55 -0700
> 
> +   When BEFORE_MARKERS, all overlays beginning/ending at POS are
> +   treated as if their front_advance/rear_advance was true. */

Please avoid this confusing style (in comments, doc strings, and the
manual).  Our preferred style is to say this instead:

  BEFORE_MARKERS non-zero means ...

or

  If BEFORE_MARKERS is non-zero, ...