I believe fit-window-to-buffer has become a bit upset and unnecessary
aggressive because of visual lines. It looks like it need a bit more
feedback from the display system to be really sure that the buffer is
entirely visible.

The attached patch is something I have used to get around the problem.
I am not sure it is the right thing but I am rather sure it does not
hurt.

Too see the problem it tries to fix just call the function with a
buffer larger than window and point below the window bottom (you have
to write a bit elisp code for that).

Of course we need a non-killing version of fit-window-to-buffer, but
for the moment this patch might be useful.