Question about (excessive?) ram usage when many overlays (with large vscroll)

[dalanicolai <dalanicolai@gmail.com>]

[-- Attachment #1.1: Type: text/plain, Size: 2199 bytes --]

I have a question about ram usage when using overlays.

So I have created `image-roll.el` for displaying documents/books (see here
<https://lists.gnu.org/archive/html/emacs-devel/2022-04/msg00975.html>).
However, I have just noticed that it uses a large amount of RAM when
viewing (or
trying to) pages in the back of 'large' books. But even if RAM usage still
looks
perfectly fine, Emacs crashes when trying to scroll to higher page numbers.

I have looked a little into it, and have found that it is a consequence of
using
large overlays. There is no problem when creating a buffer containing many
overlays, however, when trying to scroll to some overlay at the end of the
buffer,
Emacs will use huge amounts of RAM.

So I am wondering if this is 'necessary' behavior; and then why is it
necessary?
The issue occurs even when displaying the same 'empty' svg-image on each
overlay, but it occurs also when simply displaying some 'specified space' on
each overlay.

I have created a simple test file (attached to this mail) for reproducing
the
issue.From 'emacs -Q` you can simply load the file and do `M-x test`. Then
you
can type some number and press `C-c C-c` to jump to the page/overlay(number)
(use arrow up/down to scroll). The overlay width is set to 1200 pixels (via
a
global variable, so that it can easily be changed) and the width is set to
1.4
times that size (you can also comment out the 'specified space' overlay, and
uncomment the `insert-image` form). Over here, Emacs crashes when trying to
jump
to a page number of somewhere around 1750. When increasing the overlay size,
e.g. setting the width to (window-pixel-width), then RAM usage increases
much
faster, and Emacs starts to crash from jumping to page around 800.

So my question is, if it is really necessary that Emacs uses so much RAM
for just
displaying 'empty spaces'. And another question is why the maximum
`vscroll` is
(or seems to be) limited.

I have designed the book-roll like this, so that the scroll bar might be
used
for indicating/scrolling to the/some position in a document. However,
otherwise
I probably have to design it so that the buffer only contains two overlays
and
'simulate' the image-roll behavior.

[-- Attachment #1.2: Type: text/html, Size: 2417 bytes --]

[-- Attachment #2: test.el --]
[-- Type: text/x-emacs-lisp, Size: 2276 bytes --]

(require 'svg)

(setq test-page-width 1200)

(defun test-goto-page (page)
  (interactive
   (list (if current-prefix-arg
             (prefix-numeric-value current-prefix-arg)
           (read-number "Page: "))))
  (set-window-vscroll nil
                      (print (* (print page)
                                (+ (* 1.4 test-page-width)
                                   10)))
                      t))

(defun test-scroll-forward ()
  (interactive)
  (set-window-vscroll nil (+ (window-vscroll nil t) 50) t))

(defun test-scroll-backward ()
  (interactive)
  (set-window-vscroll nil (- (window-vscroll nil t) 50) t))

(define-derived-mode test-mode special-mode "Test")

(define-key test-mode-map (kbd "<down>") 'test-scroll-forward)
(define-key test-mode-map (kbd "<up>") 'test-scroll-backward)
(define-key test-mode-map (kbd "C-c C-c") 'test-goto-page)

(when (featurep 'evil)
  (evil-define-key 'motion test-mode-map "j" 'test-scroll-forward)
  (evil-define-key 'motion test-mode-map "k" 'test-scroll-backward)
  (evil-define-key 'motion test-mode-map "G" 'test-goto-page))

(defun test ()
  (interactive)
  (with-current-buffer (get-buffer-create "*test*")
    ;; (let* ((w (window-pixel-width))
    (let* ((w test-page-width)
           (h (* 1.4 w))
           (svg (svg-create w h)))

      ;; uncomment for `colored page'
      ;; (svg-rectangle svg 0 0 w h :fill "red")

      (let ((im (svg-image svg)))

        (dotimes (i 1600)

          ;; NOTE either insert `specified space' overlays
          (let ((start (point)))
            (insert " ")
            (let ((o (make-overlay start (point))))
              (overlay-put o 'display `(space . (:width (,w) :height (,h))))))

          ;; NOTE or insert an `empty' svg image
          ;; (insert-image im " ")
          (insert "\n")

          ;; for `visual clarity', insert a page separator overlay
          (let ((start (point)))
            (insert " ")
            (let ((o (make-overlay start (point))))
              (overlay-put o 'display `(space . (:width (,w) :height (,10))))
              (overlay-put o 'face `(:background "gray"))
              (insert "\n")))))
      (test-mode)
      (setq cursor-type nil)
      (switch-to-buffer (current-buffer))
      (goto-char (point-min)))))