IMO: There is more than one issue at the same time:

1. emacs should not abort when having debug-on-error on
2. close-display-connection should not close both frames.
3. the X Glyph issue should not happen. Is it totally normal closing X-remote frames.

BTW. After inspecting the gdb log. The issue #1 happens within the elisp
function debug. inside that function there is this line.

--8<---------------cut here---------------start------------->8---
(kill-emacs -1)))
--8<---------------cut here---------------end--------------->8---

That in this particular case should not be called. 

Best Regards