Can't we instead have a database of these messages, like we do with
the "password" prompts?

I am not familiar with all grep options but I saw that:

‘-I’
Process a binary file as if it did not contain matching data; this
is equivalent to the ‘--binary-files=without-match’ option.

Just tested

(setq xref-search-program-alist '((grep . "xargs -0 grep <C> --null -snHE -I -e <R>"))

and it works fine on my side.

Matthias