Seems the problem is related to syntax highlighting parsers in diffs.

I don't need syntax highlighting at the cost of slowness and polluted recentf history.