I think the suggestion to use a custom variable (or just use byte-count-to-string) is a good one, but AFAICS the argument to file-size-human-readable should be bytes, but proced gets rss and vsize from 'process-attributes' which uses kilobytes.  So we're still stuck with having to choose one of:

(funcall #'byte-count-to-string-function (* 1000 kilobytes))

(funcall #'byte-count-to-string-function (* 1024 kilobytes))

I'm not really familiar with c, so I can't tell from process.c