This is the reason I only included file bound buffers initially in switch-to-buffer. But as Andrii noted, there are other interesting buffers as well, such as xref etc. 

Maybe we can add some logic to filter out more of the unrelated buffers?

Theo