I think I found the solution: set_invocation_vars requires Vexec_path to be initialized correctly, via init_callproc_1. Somehow, mine ends up being something other than $PATH. If I set init_callproc_1 up with a double_run_guard the same way set_invocation_var is and call it from pdumper_load, it results in a correctly-set Vexec_path variable and a correctly-set Vinvocation_directory, as well.

This is really curious, but I guess it would explain why the invocation directory variable is empty! What it doesn't explain is, why does this work for you? Maybe your $PATH variable is set up differently? (Though I don't see where exec-path would be set otherwise...)