Not selecting the WoMan buffer explicitly creates this problem starting from these lines in the woman-find-file function:

That's because Man-build-section-alist does (goto-char (point-min)) in which ever the selected buffer is.

display-buffer does not select the buffer.. so that explicit switch-to-buffer call had to be put there.

In my opinion, if we do not want to do switch-to-buffer calls in WoMan-find-buffer, a more elegant way would be to wrap the below lines in woman-find-file

with a (with-current-buffer WOMAN-BUFFER ..) form.

WDYT?