Font size of line numbers by `linum-mode' follow `text-scale-adjust', though
the size of left margin doesn't change.
(nlinum also has the same problem)

The possible solutions are,


(1) To change the width of left margin for line numbers.

[attached patch]


(2) To fix the font size of line numbers.

(set-face-attribute 'linum nil :height (face-attribute 'linum :height nil t))


Solution (2) may need more in case of multi frames with different font size.


Regards,
Shigeru