As for shell, file name completion only seems to breaks down for me when
completing a relative file name argument to the cd command in
particular:

0. emacs -Q
1. M-x shell RET
2. cd /tmp/
   [shell doesn't seem to support dynamic cd arguments such as
    "$(mktemp -d)".]
3. mkdir dir\ one dir\ two RET
4. cd /tmp/dir TAB
   [Completes to "/tmp/dir\ ".]
5. TAB
   [Offers "dir\ one/" and "dir\ two/" as completion candidates.]
6. o TAB
   [Completes to "/tmp/dir\ one/".]

Replacing cd with ls (or pretty much any other string, let alone valid
command) in step 4 exhibits the same correct behaviour.  Doing so in
OP's recipe (which tries to complete relative file names), however, does
not seem to struggle in the same way as cd completion:

Steps 0-3 as before.
4. ls dir TAB
   [Completes to "dir\ ".]
5. TAB
   [Offers "dir\ one/" and "dir\ two/" as completion candidates.]
5. o TAB
   [Completes to "/tmp/dir\ one/".]

I'm not familiar with the workings of shell, but it seems like it's
trying to be smart about cd completion and falling a bit short.

-- 
Basil

In GNU Emacs 27.0.50 (build 4, x86_64-pc-linux-gnu, X toolkit, Xaw3d scroll bars)
 of 2018-05-27 built on thunk
Repository revision: 0f48d18fd2a30f29cc3592a835d2a2254c9b0afb
Windowing system distributor 'The X.Org Foundation', version 11.0.11906000
System Description: Debian GNU/Linux buster/sid

"${SHELL}" --version
GNU bash, version 4.4.19(1)-release (x86_64-pc-linux-gnu)