I forgot to say, the way I'm spawning vterm is just effectively:

(let ((default-directory "sshx:<host>:.")) (vterm))