bug#46326: 27.1.50; Excessive memory allocations with minibuffer-with-setup-hook

[mail@daniel-mendler.de]

Dear Emacs developers,

I have an issue on 27.1.50 with excessive memory allocations when using
minibuffer-with-setup-hook with large closures and :append. This issue 
has been
observed in my Consult package, which is similar to Counsel and which 
provides
commands based on completing-read. Naturally I have to perform some 
setup in the
minibuffer-setup-hook. Most of the memory allocations are due to 
add-hook and in
particular the sort function. The sort functions is called because of 
:append
which sets a hook priority. The reason seems to be that large closures 
are
copied, but I didn't fully investigate the reasons for the issue.

Example profile:

     140,068,687  92%       - consult-buffer
     140,068,687  92%        - consult--buffer
     140,068,687  92%         - consult--multi
     138,933,527  91%          - consult--read
     138,933,527  91%           - apply
     138,933,527  91%            - #<subr consult--read>
      99,916,347  65%             - consult--with-async-1
      99,916,347  65%              - #<compiled 0x3e02774d5b1a3f7>
      99,912,203  65%               - consult--with-preview-1
      80,735,735  53%                - #<compiled -0xbfb98ad3a8adf1c>
      80,735,735  53%                 - completing-read
      80,735,735  53%                  - selectrum-completing-read
      62,909,765  41%                   - selectrum-read
>>>  58,826,659  38%                    + add-hook
       1,077,206   0%                    + 
selectrum--minibuffer-post-command-hook
       ....
>>>  19,176,468  12%                + add-hook
>>>  39,017,180  25%             + add-hook

The issue can be mitigated by using a modified version of
minibuffer-with-setup-hook, where I am creating a symbol and fsetting 
instead of
adding a lambda directly via add-hook. This technique is also used in
set-transient-map in order to avoid problems with letrec-closures and
add-hook/remove-hook.

     (defmacro consult--minibuffer-with-setup-hook (fun &rest body)
       (declare (indent 1) (debug t))
       (let ((hook (make-symbol "hook"))
             (append))
         (when (eq (car-safe fun) :append)
           (setq append '(t) fun (cadr fun)))
         `(let ((,hook (make-symbol "consult--minibuffer-setup")))
            (fset ,hook (lambda ()
                          (remove-hook 'minibuffer-setup-hook ,hook)
                          (funcall ,fun)))
            (unwind-protect
                (progn
                  (add-hook 'minibuffer-setup-hook ,hook ,@append)
                  ,@body)
              (remove-hook 'minibuffer-setup-hook ,hook)))))

See the issue https://github.com/minad/consult/issues/193 for reference.

Maybe there exists an alternative solution to the problem and the 
problem should
be fixed on a different level than by replacing the 
minibuffer-with-setup-hook
macro?

Daniel Mendler

In GNU Emacs 27.1.50 (build 1, x86_64-pc-linux-gnu, GTK+ Version 3.24.5, 
cairo version 1.16.0) of 2020-10-24
Repository revision: c847d5998f588dbf3eca5ea1ec573a2d64a97607
Repository branch: emacs-27
Windowing system distributor 'The X.Org Foundation', version 
11.0.12004000
System Description: Debian GNU/Linux 10 (buster)