Sorry, found this bug only now. Did some profile measurements and
reported it here:

https://debbugs.gnu.org/cgi/bugreport.cgi?bug=26402

Let me know if I can be useful. Seems like quite an big issue.