This is on Windows, and running Emacs in-place after building from trunk (no installation).

If you have several VERSION-xxxxxxxx directories under native-lisp/ and do 

M-x native-compile-prune-cache <RET>

this deletes any .eln files in the VERSION-xxxxxxxx directories of the stale caches. Their preloaded/ subdirectories and content are not affected.

Presumably this should skip the last entry in native-comp-eln-load-path, which is documented as being "assumed to be the system one".