The problem is that various popular packages make extensive use of the (display-graphic-p) function/predicate to query frame capabilities. For example, a theme package may query (display-graphic-p) and assign gui or tty colors accordingly.
Which packages are those, how can one recreate the issue you are seeing in an emacs -Q session (for the case of emacsclient, I am aware that emacs -Q does not help, so one would need to temporarily move their ~/.emacs.d/ dir to something else, and use a minimal config that just starts the server) after loading those packages?
--