(Aha, updating just xterm fixes the problem).

When locales don't work, I can't enter umlauts at all - they get replaced by a
"generic" character that isn't an umlaut, even when checked using a hexdump
tool.

So not only couldn't I see the umlauts, I couldn't type them blindly either.