I changed the affected faces to use inverse-video, then hl-line only modified the foreground color of the text, which is better than before.