Sorry about the confusion.

That part of the display appears to use the same font size as the
buffer after text-scale has been enabled.

> Anyway, the change you suggest is basically going back to what we had
> before fixing bug#28855, so I don't think we can make such a change,
> at least not literally.  We need something more complex to account for
> the use cases described in that bug.

Yes, I had a feeling what I posted was not correct, because it seemed
like the "obvious" thing, and so, would've been there already if
correct.

I've just dug through the Elisp manual a little and I think (elisp)
Specified Space is precisely what's needed here.  I've tried the
following:

modified   lisp/ruler-mode.el
@@ -632,12 +632,7 @@ ruler-mode-ruler
   (let* ((w (ruler-mode-text-scaled-window-width))
          (m (window-margins))
          (f (window-fringes))
-         (i (if display-line-numbers
-                ;; FIXME: ruler-mode relies on I being an integer, so
-                ;; the column numbers might be slightly off if the
-                ;; line-number face is customized.
-                (round (line-number-display-width 'columns))
-              0))
+         (i 0)
          (j (ruler-mode-text-scaled-window-hscroll))
          ;; Setup the scrollbar, fringes, and margins areas.
          (lf (ruler-mode-space
@@ -739,7 +734,13 @@ ruler-mode-ruler
       (setq i (1+ i)
             j (1+ j)))
 
-    (let ((ruler-str (concat ruler))
+    (let ((ruler-indent
+           (if (not display-line-numbers)
+               ""
+             (propertize " " 'display
+                         `(space :width
+                                 (,(- (line-number-display-width t) 1))))))
+          (ruler-str (concat ruler))
           (len (length ruler)))
       (add-text-properties 0 len ruler-wide-props ruler-str)
       (dolist (p (nreverse props))
@@ -747,13 +748,14 @@ ruler-mode-ruler
 
       ;; Return the ruler propertized string.  Using list here,
       ;; instead of concat visually separate the different areas.
-      (if (nth 2 (window-fringes))
-          ;; fringes outside margins.
-          (list "" (and (eq 'left sbvt) sb) lf lm
-                ruler-str rm rf (and (eq 'right sbvt) sb))
-        ;; fringes inside margins.
-        (list "" (and (eq 'left sbvt) sb) lm lf
-              ruler-str rf rm (and (eq 'right sbvt) sb))))))
+      (let ((ruler-str-ind (concat ruler-indent ruler-str)))
+            (if (nth 2 (window-fringes))
+            ;; fringes outside margins.
+            (list "" (and (eq 'left sbvt) sb) lf lm
+                  ruler-str-ind rm rf (and (eq 'right sbvt) sb))
+          ;; fringes inside margins.
+          (list "" (and (eq 'left sbvt) sb) lm lf
+                ruler-str-ind rf rm (and (eq 'right sbvt) sb)))))))
 
 (provide 'ruler-mode)
 
This is almost perfect. The (- ... 1) in the :width pixel spec is
obviously a hack (and breaks this patch in emacs -nw).  I have no idea
why an extra pixel is inserted in GUI mode.  If you apply the above but
remove (- ... 1), you'll see the following: