Ok! Here's my first try at this. I ended up skipping the check on
DEFAULT-DIRECTORY since as you mentioned, its value is used with
expand-file-name itself. In the other case, if default-directory is
picked up, then I checked the value of that variable.