bug#46908: 28.0.50; Dozens of spurious markers in buffer-undo-list

bug#46908: 28.0.50; Dozens of spurious markers in buffer-undo-list

[Michael Heerdegen]

Hello,

after using `undo', `buffer-undo-list' often contains entries
referencing several or dozens of markers (seemingly unrelated to
overlays or such; I dunno from where they originate).  I know this
happens in current Emacs master, and I also found the problem in the
27.1 coming with Debian testing, build from 2020-11-08.  I tried in X.

I measure the count of different marker objects in a buffer's
`buffer-undo-list' with this function:

#+begin_src emacs-lisp
(defun count-markers-in-buffer-undo-list ()
  "Message number of (different) markers in `buffer-undo-list'."
  (interactive)
  (require 'cl-lib)
  (message "%d" (length
                 (cl-delete-duplicates
                  (seq-filter #'markerp (flatten-tree (copy-tree buffer-undo-list)))
                  :test #'eq))))
#+end_src

To reproduce the problem, open a fresh emacs -Q, and in *scratch*,
something like this should do it:

"C-5 C-0 a, RET, C-SPC, C-p, C-SPC, C-n, M-w,
C-y, C-y, C-y";

this recipe is by Yuan Fu, who first reported the problem in emacs-help,
and who is CC'd.  Then he calls the above command and gets a number much
larger than 1.

I can reproduce the problem without copy-and-paste, I just hold down the
"a" key and insert some random space chars in between, then undo
everything.  Sometimes I need to repeat once or twice to get a number
larger than "1", but very soon I see numbers like "40" which definitely
look pathological.  For me the appearance of the issue seems to behave a
bit randomly.


Thanks in advance,

Michael.



In GNU Emacs 28.0.50 (build 6, x86_64-pc-linux-gnu, GTK+ Version 3.24.24, cairo version 1.16.0)
 of 2021-03-03 built on drachen
Repository revision: b28b8d83ac882415555fe02b73a26542c9fca781
Repository branch: master
Windowing system distributor 'The X.Org Foundation', version 11.0.12010000
System Description: Debian GNU/Linux bullseye/sid

Configured features:
CAIRO DBUS FREETYPE GIF GLIB GMP GNUTLS GSETTINGS HARFBUZZ JPEG JSON
LCMS2 LIBSELINUX LIBXML2 MODULES NOTIFY INOTIFY PDUMPER PNG RSVG SOUND
THREADS TIFF TOOLKIT_SCROLL_BARS X11 XDBE XIM XPM GTK3 ZLIB

Important settings:
  value of $LC_ALL: de_DE.utf8
  value of $LC_COLLATE: C
  value of $LC_TIME: C
  value of $LANG: de_DE.utf8
  locale-coding-system: utf-8-unix

bug#46908: 28.0.50; Dozens of spurious markers in buffer-undo-list

[martin rudalics]

 > this recipe is by Yuan Fu, who first reported the problem in emacs-help,
 > and who is CC'd.  Then he calls the above command and gets a number much
 > larger than 1.

IIRC the thread on emacs-help mentioned repeated occurrences of setting
the window-point marker.  Is that still (one of) the main issue(s) or
does the problem happen independently of that?.

martin

bug#46908: 28.0.50; Dozens of spurious markers in buffer-undo-list

[Michael Heerdegen]

martin rudalics <rudalics@gmx.at> writes:

> IIRC the thread on emacs-help mentioned repeated occurrences of setting
> the window-point marker.  Is that still (one of) the main issue(s) or
> does the problem happen independently of that?.

That was guessing, because winner-mode had been mentioned.  But the
issue seems to be unrelated to winner-mode.

Independently from the window-point marker?  I don't know if there is a
relation to the window-point marker, probably not, because, if there is,
I guess we would not see dozens of _different_ markers.  These could be
copies of the window-point marker created somewhere, I have no clue.

Michael.

bug#46908: 28.0.50; Dozens of spurious markers in buffer-undo-list

[martin rudalics]

 > Independently from the window-point marker?  I don't know if there is a
 > relation to the window-point marker, probably not, because, if there is,
 > I guess we would not see dozens of _different_ markers.  These could be
 > copies of the window-point marker created somewhere, I have no clue.

When I read the discussion on emacs-help I tried with `buffer-undo-list'
nil and doing some window modifications but didn't find a smoking gun.
So I'm afraid you have to look into your undo lists more closely to find
out whether some `with-silent-modifications' are missing.  Maybe
'winner-mode' and its window configuration and active region handling
can give a clue.

martin

bug#46908: 28.0.50; Dozens of spurious markers in buffer-undo-list

[Michael Heerdegen]

martin rudalics <rudalics@gmx.at> writes:

> When I read the discussion on emacs-help I tried with `buffer-undo-list'
> nil and doing some window modifications but didn't find a smoking gun.

AFAIK window modifications are not necessary to reproduce the issue,
probably they are completely unrelated.

Did you try to reproduce the issue with the recipe?  It is complete.

> So I'm afraid you have to look into your undo lists more closely to find
> out whether some `with-silent-modifications' are missing.  Maybe
> 'winner-mode' and its window configuration and active region handling
> can give a clue.

Sorry for the confusion: We did not turn winner-mode on in the recipe,
please just ignore that part of the discussion, that was a false lead.
You just need insert and undo to reproduce.  Please try.  emacs -Q, just
editing commands plus undo produce the zombi markers.

Regards,

Michael.

bug#46908: 28.0.50; Dozens of spurious markers in buffer-undo-list

[martin rudalics]

 >> When I read the discussion on emacs-help I tried with `buffer-undo-list'
 >> nil and doing some window modifications but didn't find a smoking gun.
 >
 > AFAIK window modifications are not necessary to reproduce the issue,
 > probably they are completely unrelated.

Maybe.  But I don't know of any packages that are much interested in
reading or manipulating `window-point'.  So if `window-point' is
involved in at least one scenario that reproduces the problem, it might
help to trace that occurrence back to its origins.

 > Did you try to reproduce the issue with the recipe?  It is complete.

I did not because I just believe you that your recipe is valid.  The
problem is that you have to trace back those marker entries back to
where they have been created in such abundance.  And I have no idea
how to do that.

 > Sorry for the confusion:

You did not confuse me.

 > We did not turn winner-mode on in the recipe,
 > please just ignore that part of the discussion, that was a false lead.

If turning on `winner-mode' _can_ create many `window-point' marker
entries, it's not necessarily a false lead.  It might help to locate the
real culprit with one important restriction: You'd have to look into
clients that either deal with `window-point' directly or call something
that does deal with `window-point' only.

 > You just need insert and undo to reproduce.  Please try.  emacs -Q, just
 > editing commands plus undo produce the zombi markers.

I tried it now.  The only thing I can say is that with Emacs 27 it
usually returns 0 or 1 here while with master it returns 10.  So maybe
you should try (after verifying what I found out) to trace the recent
changes in master.

martin

bug#46908: 28.0.50; Dozens of spurious markers in buffer-undo-list

[martin rudalics]

 > I tried it now.  The only thing I can say is that with Emacs 27 it
 > usually returns 0 or 1 here while with master it returns 10.  So maybe
 > you should try (after verifying what I found out) to trace the recent
 > changes in master.

The "it" above is `count-markers-in-buffer-undo-list'.  And with "trace"
I really meant "bisect".  But make sure to check first what I said about
the Emacs 27 behavior - YMMV.

martin

bug#46908: 28.0.50; Dozens of spurious markers in buffer-undo-list

[Michael Heerdegen]

martin rudalics <rudalics@gmx.at> writes:

> > I tried it now.  The only thing I can say is that with Emacs 27 it
> > usually returns 0 or 1 here while with master it returns 10.  So maybe
> > you should try (after verifying what I found out) to trace the recent
> > changes in master.
>
> The "it" above is `count-markers-in-buffer-undo-list'.  And with "trace"
> I really meant "bisect".  But make sure to check first what I said about
> the Emacs 27 behavior - YMMV.

I cannot confirm that, no.  I can reproduce the issue in a newly built
Emacs 27 as well as in the 27 that comes with Debian.  "It" again being
`count-markers-in-buffer-undo-list'.

My question is what amount of that is pathological.  A lot of code
creates markers "on the fly", a lot, e.g. jit-lock or yank (push-mark)
or display-buffer.  When undo is performed before gc is run, they end in
buffer-undo-list.

Does making a marker point to nil with `set-marker' then make it
suitable for garbage collection?  I guess not.

My observation: if I set undo-buffer-list to nil after reproducing the
issue, `count-markers-in-buffer-undo-list' reports "0" afterwards.  So I
wonder whether what we see is maybe normal business and an unfortunate
side effect of undo recording marker positions and gc running not often
enough to prevent these accidents?

I can continue testing with emacs-26 if you want.

Regards,

Michael.

bug#46908: 28.0.50; Dozens of spurious markers in buffer-undo-list

[Eli Zaretskii]

> From: Michael Heerdegen <michael_heerdegen@web.de>
> Date: Sat, 06 Mar 2021 01:51:00 +0100
> Cc: 46908@debbugs.gnu.org, Yuan Fu <casouri@gmail.com>
> 
> My question is what amount of that is pathological.  A lot of code
> creates markers "on the fly", a lot, e.g. jit-lock or yank (push-mark)
> or display-buffer.  When undo is performed before gc is run, they end in
> buffer-undo-list.
> 
> Does making a marker point to nil with `set-marker' then make it
> suitable for garbage collection?  I guess not.

GC compacts the buffer's undo-list, and that is supposed to remove
those entries.  Doesn't it?

bug#46908: 28.0.50; Dozens of spurious markers in buffer-undo-list

[Michael Heerdegen]

Eli Zaretskii <eliz@gnu.org> writes:

> > Does making a marker point to nil with `set-marker' then make it
> > suitable for garbage collection?  I guess not.
>
> GC compacts the buffer's undo-list, and that is supposed to remove
> those entries.  Doesn't it?

Maybe it does that.

What I can say that gc removes a significant amount of markers, but that
also a lot are left after gc.  And the positions of the markers that are
left are numbers (not nil).

Michael.

bug#46908: 28.0.50; Dozens of spurious markers in buffer-undo-list

[martin rudalics]

 > My observation: if I set undo-buffer-list to nil after reproducing the
 > issue, `count-markers-in-buffer-undo-list' reports "0" afterwards.

What else did you expect?  Or did you mean in all subsequent runs of
your scenario?

 > So I
 > wonder whether what we see is maybe normal business and an unfortunate
 > side effect of undo recording marker positions and gc running not often
 > enough to prevent these accidents?

I'm not sure what you mean.  Here, when trying your scenario, I do set
`buffer-undo-list' to nil first, so the initial insertion is not
recorded and run `count-markers-in-buffer-undo-list' from a globally set
key so the minibuffer is not involved.  And meanwhile I cannot reproduce
your scenario with Emacs 28 either, I get zero invariably.

martin

bug#46908: 28.0.50; Dozens of spurious markers in buffer-undo-list

[Michael Heerdegen]

martin rudalics <rudalics@gmx.at> writes:

> I'm not sure what you mean.  Here, when trying your scenario, I do set
> `buffer-undo-list' to nil first, so the initial insertion is not
> recorded and run `count-markers-in-buffer-undo-list' from a globally set
> key so the minibuffer is not involved.  And meanwhile I cannot reproduce
> your scenario with Emacs 28 either, I get zero invariably.

Just now I tried exactly that, exactly that way.  Using only
self-insert-command invocations and C-SPC, M-w and C-y, I was able to
get to values of more than 20 in 15 seconds or so with the first
attempt.

Michael.

bug#46908: 28.0.50; Dozens of spurious markers in buffer-undo-list

[Michael Heerdegen]

Michael Heerdegen <michael_heerdegen@web.de> writes:

> I cannot confirm that, no.  I can reproduce the issue in a newly built
> Emacs 27 as well as in the 27 that comes with Debian.  "It" again being
> `count-markers-in-buffer-undo-list'.

The issue disappears (marker count stays constantly less than 3) if I
comment out these two expressions in `push-mark':

  (add-to-history 'mark-ring (copy-marker (mark-marker)) mark-ring-max t)

  (add-to-history
    'global-mark-ring (copy-marker (mark-marker)) global-mark-ring-max t)

I think we really have a problem here: when those newly created markers
are thrown out of the rings, they are not invalidated, and are also
still referenced by buffer-undo-list, so they will live forever, AFAIU.

Michael.

bug#46908: 28.0.50; Dozens of spurious markers in buffer-undo-list

[martin rudalics]

 >> I cannot confirm that, no.  I can reproduce the issue in a newly built
 >> Emacs 27 as well as in the 27 that comes with Debian.  "It" again being
 >> `count-markers-in-buffer-undo-list'.
 >
 > The issue disappears (marker count stays constantly less than 3) if I
 > comment out these two expressions in `push-mark':
 >
 >    (add-to-history 'mark-ring (copy-marker (mark-marker)) mark-ring-max t)
 >
 >    (add-to-history
 >      'global-mark-ring (copy-marker (mark-marker)) global-mark-ring-max t)
 >
 > I think we really have a problem here: when those newly created markers
 > are thrown out of the rings, they are not invalidated, and are also
 > still referenced by buffer-undo-list, so they will live forever, AFAIU.

So you mean that we really have two problems here, both caused by
`push-mark'?  IIUC a marker is "thrown out of the rings" when its buffer
gets deleted or a marker is pushed on a "full" ring.  I suppose that
"invalidating" markers in these cases should be safe.  But how comes
that `add-to-history' affects `buffer-undo-list'?

martin

bug#46908: 28.0.50; Dozens of spurious markers in buffer-undo-list

[Michael Heerdegen]

martin rudalics <rudalics@gmx.at> writes:

> So you mean that we really have two problems here, both caused by
> `push-mark'?  IIUC a marker is "thrown out of the rings" when its buffer
> gets deleted or a marker is pushed on a "full" ring.  I suppose that
> "invalidating" markers in these cases should be safe.

But actually that doesn't help, markers pointing to nowhere are not
removed from buffer-undo-list by garbage-collect.

> But how comes that `add-to-history' affects `buffer-undo-list'?

`add-to-history' has no relevance here, no.  There mere existence of a
marker in any buffer does, IIUC, cause it to be referenced by
buffer-undo-list sooner or later, because undoing and redoing inside
text containing a marker is not always without ambiguity regarding
marker positioning, so the undo system adds hints to buffer-undo-list
how these markers must be placed/repositioned.

Michael.

bug#46908: 28.0.50; Dozens of spurious markers in buffer-undo-list

[Michael Heerdegen]

Michael Heerdegen <michael_heerdegen@web.de> writes:

> > But how comes that `add-to-history' affects `buffer-undo-list'?
>
> `add-to-history' has no relevance here, no.  There mere existence of a
> marker in any buffer does, IIUC, cause it to be referenced by
> buffer-undo-list sooner or later, because undoing and redoing inside
> text containing a marker is not always without ambiguity regarding
> marker positioning, so the undo system adds hints to buffer-undo-list
> how these markers must be placed/repositioned.

I did some more experimentation (repeated copy, paste, undo and
garbage-collect in *scratch*), and I saw that the number of markers
living in buffer-undo-list is not monotonously increasing.  After some
time, the amount settles between 20 and 45 or so.  And stays in that
range.  Giving that a large, but still limited number of copies of the
mark marker is stored in the mark rings, that amount of markers isn't a
surprise, and if the number is not constantly growing,
everything...works as expected so far I think.  Correct?

Michael.

bug#46908: 28.0.50; Dozens of spurious markers in buffer-undo-list

[Lars Ingebrigtsen]

Michael Heerdegen <michael_heerdegen@web.de> writes:

> I did some more experimentation (repeated copy, paste, undo and
> garbage-collect in *scratch*), and I saw that the number of markers
> living in buffer-undo-list is not monotonously increasing.  After some
> time, the amount settles between 20 and 45 or so.  And stays in that
> range.  Giving that a large, but still limited number of copies of the
> mark marker is stored in the mark rings, that amount of markers isn't a
> surprise, and if the number is not constantly growing,
> everything...works as expected so far I think.  Correct?

I just lightly skimmed this thread, but does this mean that there's
nothing to fix here?

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#46908: 28.0.50; Dozens of spurious markers in buffer-undo-list

[Michael Heerdegen]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> > Giving that a large, but still limited number of copies of the
> > mark marker is stored in the mark rings, that amount of markers isn't a
> > surprise, and if the number is not constantly growing,
> > everything...works as expected so far I think.  Correct?
>
> I just lightly skimmed this thread, but does this mean that there's
> nothing to fix here?

Not unlikely.

Ok, let's close this one, I don't think this will lead to
something.

Michael.

bug#46908: 28.0.50; Dozens of spurious markers in buffer-undo-list

[martin rudalics]

 > `add-to-history' has no relevance here, no.  There mere existence of a
 > marker in any buffer does, IIUC, cause it to be referenced by
 > buffer-undo-list sooner or later, because undoing and redoing inside
 > text containing a marker is not always without ambiguity regarding
 > marker positioning, so the undo system adds hints to buffer-undo-list
 > how these markers must be placed/repositioned.

Suppose I put a marker into a buffer and do ten distinct modifications
of the buffer, each modifying that marker's position.  So I get ten
additional entries in the undo list.  Does this mean I get ten copies of
that marker too?

martin

bug#46908: 28.0.50; Dozens of spurious markers in buffer-undo-list

[Michael Heerdegen]

martin rudalics <rudalics@gmx.at> writes:

> Suppose I put a marker into a buffer and do ten distinct modifications
> of the buffer, each modifying that marker's position.  So I get ten
> additional entries in the undo list.  Does this mean I get ten copies
> of that marker too?

As far as I understand - of course not!  When undo needs to restore
previous positions of that marker, it only can do that when the entries
in buffer-undo-list refer to the original marker.

`push-mark' does copy the mark marker, OTOH, it must, because a mark
ring containing the mark marker 16 times would be nonsense.

Michael.

bug#46908: 28.0.50; Dozens of spurious markers in buffer-undo-list

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > But actually that doesn't help, markers pointing to nowhere are not
  > removed from buffer-undo-list by garbage-collect.

Does undo ever need to relocate a marker that points to nowhere?  I
think it doesn't, so I think it would be ok for GC to delete them.

Undo could also delete them.

-- 
Dr Richard Stallman
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

bug#46908: 28.0.50; Dozens of spurious markers in buffer-undo-list

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > I think we really have a problem here: when those newly created markers
  > are thrown out of the rings, they are not invalidated, and are also
  > still referenced by buffer-undo-list, so they will live forever, AFAIU.

I am getting the feeling that references from buffer-undo-list should
not keep markers alive.  That they should be some sort of weak reference.


-- 
Dr Richard Stallman
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

bug#46908: 28.0.50; Dozens of spurious markers in buffer-undo-list

[Michael Heerdegen]

Richard Stallman <rms@gnu.org> writes:

> I am getting the feeling that references from buffer-undo-list should
> not keep markers alive.  That they should be some sort of weak
> reference.

Could well be that it already works like this, the result of my
experiments went this way.  I currently just don't understand how that
stuff works.

Regards,

Michael.