1. Do a search that has several matches in one buffer, e.g. using
dired-do-find-regexp.
2. Press `r' in the *xref* buffer, to initiate replacement.  Input `.*'
and `abcd', for instance.  This value of FROM is important.
3. Replace a couple then undo that (press y, y, u, u).
4. Try agreeing with the subsequent prompts.  The replacements performed
then will be wrong.

I've tried to come up with a patch but stopped short of really delving
into the code of perform-replace. Which seems really necessary at this
point. Attaching what I already have.

The big problem is that perform-replace does not consistently use
replace-re-search-function. With the new undo code, it's became worse.
After you press `u', it seems to switch to moving around the saved match
data stack and using a plain looking-at, without checking with
isearch-filter-predicate.