Steps to reproduce:

1) Have a frame with such width and height that when there are 4 equal
windows inside it, a `pop-to-window' invocation reuses an existing
window.

2) Start with just 1 window, do C-x 3, then C-x 2 in both resulting
windows.

3) Open a VC-backed file in one of the windows, make a tiny change,
save, do `vc-diff'. Diff will open in one of the existing windows, and
it will be shrunk to the height of the buffer.

4) Quit the diff buffer with `q'. See that the window height stays
shrunk.

In a similar scenario, open a file from a repository with tiny history
(1 commit or so), do `vc-print-log', observe the same behavior.
Thankfully, this is a much more rare occurrence.

Simple patch attached.

Commands messing up existing window configuration is one of my top
Emacs annoyances, and AFAIK it confuses the new users, too.
Maybe nil check for (window-prev-buffers) should be instead included in
`shrink-window-if-larger-than-buffer', with a way to override it?