I think the suggestion to use a custom variable (or just use byte-count-to-string) is a good one, but AFAICS the argument to file-size-human-readable should be bytes, but proced gets rss and vsize from 'process-attributes' which uses kilobytes. So we're still stuck with having to choose one of:
(funcall #'byte-count-to-string-function (* 1000 kilobytes))
(funcall #'byte-count-to-string-function (* 1024 kilobytes))
I'm not really familiar with c, so I can't tell from process.c