Many of the buffer locals (in particular I think it's the
SYMBOL_FORWARDED types --- my terminoly may be off), appear to be O(n)
to bind, where n is the number of live buffers.

I tried to see if I can optimize that binding process directly, but
couldn't figure out a way to do it without changing current behavior,
so I opted to just optimize ido-switch-buffer directly.

After this patch, the performance is quite bearable at ~6000 buffers,
but still not ideal.