> I use `insert-file-contents-literally' because the non-literally
> counterpart is too slow (about twice as much apparently) as it does a
> bunch of stuff in addition to simply populate the buffer.
> Unfortunately, one of these things is to decode the buffer.

For better performance I restrict the size of inserted file by giving to
`insert-file-contents' a small value of args BEG and END.  For example,
to automatically detect encodings of files for diff I use such customization: