Steps to reproduce:

 1. M-: (ido-mode 'buffer)

 2. Note that C-x C-f is *not* bound to any ido stuff.

 3. M-: (ido-mode 'both)

 4. Note that C-x C-f *is* bound to ido stuff.

 5. M-: (ido-mode 'buffer)

 6. Note that C-x C-f is *still* bound to ido stuff.

Here's one solution: