Maybe there's some other way to tell if a gui window is showing / waiting for input? Then as a workaround I could just skip checking the clipboard.
I looked at some of the internals - as Eli said there's an event loop for the x dialog that also checks the existing Emacs timers, which in this case would be checking the x selection and starting another timer.
I'm not sure why that would cause a hang though, if the x selection returns immediately.
Thanks again for your help in tracking this down - I'll keep tracing through the code to see if I can figure out what's going on, and why menu-or-popup-active-p doesn't help.