​I have written a Python script that can be used to get this answer.  (It uses the Python-to-Objective-C bridge library that exposes the relevant Mac APIs to Python)​.  The script  takes two arguments, X and Y screen pixel coordinates, gets a list of all visible application windows in Z-order and then prints the application name of the first window that intersects the (X,Y) position, if any.  If that window is an Emacs frame, we have a match and otherwise, we don't.

This is a short script that really needs to be recoded into an Objective-C function that can be exposed to Emacs Lisp.  If anyone who is familiar with macOS Objective-C code would want to work with me on this, email me and I'll provide the Python code.  It is only about 28 lines.  -- Bob