I can't reproduce that with emacs -Q where my load-history after startup
has just 139 entries rather than about 600 with my configured emacs.
For the same reason, I haven't been able to check if the problem is also
there with emacs 27 because my ~/.emacs is full of emacs 28 stuff.

Daniel, I've Cc'ed you just because you might be able to find some
workaround in marginalia.
    

In GNU Emacs 28.0.50 (build 1, x86_64-pc-linux-gnu, GTK+ Version 3.24.29, cairo version 1.17.4)
 of 2021-06-04 built on thinkpad-t440p
Repository revision: 7b50ed553faa6de6d51bf07d12d106ca61ab3ac4
Repository branch: master
Windowing system distributor 'The X.Org Foundation', version 11.0.12101001
System Description: Arch Linux

Configured using:
 'configure --with-modules'

Configured features:
ACL CAIRO DBUS FREETYPE GIF GLIB GMP GNUTLS GPM GSETTINGS HARFBUZZ JPEG
JSON LCMS2 LIBOTF LIBSYSTEMD LIBXML2 M17N_FLT MODULES NOTIFY INOTIFY
PDUMPER PNG RSVG SECCOMP SOUND THREADS TIFF TOOLKIT_SCROLL_BARS X11 XDBE
XIM XPM GTK3 ZLIB

Important settings:
  value of $LC_MONETARY: de_DE.utf8
  value of $LC_NUMERIC: de_DE.utf8
  value of $LC_TIME: de_DE.utf8
  value of $LANG: en_US.UTF-8
  locale-coding-system: utf-8-unix

Major mode: Text

Minor modes in effect:
  global-aggressive-indent-mode: t
  diredfl-global-mode: t
  dynamic-completion-mode: t
  which-key-mode: t
  company-posframe-mode: t
  global-company-mode: t
  yas-global-mode: t
  yas-minor-mode: t
  global-git-commit-mode: t
  magit-auto-revert-mode: t
  vertico-mode: t
  marginalia-mode: t
  override-global-mode: t
  minibuffer-depth-indicate-mode: t
  recentf-mode: t
  pixel-scroll-mode: t
  save-place-mode: t
  savehist-mode: t
  show-paren-mode: t
  shell-dirtrack-mode: t
  tooltip-mode: t
  global-eldoc-mode: t
  electric-indent-mode: t
  mouse-wheel-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  blink-cursor-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  column-number-mode: t
  line-number-mode: t
  transient-mark-mode: t

Load-path shadows:
~/Repos/el/mu/build/mu4e/mu4e hides ~/Repos/el/mu/mu4e/mu4e
~/Repos/el/mu/build/mu4e/mu4e-main hides ~/Repos/el/mu/mu4e/mu4e-main
~/Repos/el/mu/build/mu4e/mu4e-view hides ~/Repos/el/mu/mu4e/mu4e-view
~/Repos/el/mu/build/mu4e/mu4e-org hides ~/Repos/el/mu/mu4e/mu4e-org
~/Repos/el/mu/build/mu4e/mu4e-lists hides ~/Repos/el/mu/mu4e/mu4e-lists
~/Repos/el/mu/build/mu4e/mu4e-actions hides ~/Repos/el/mu/mu4e/mu4e-actions
~/Repos/el/mu/build/mu4e/mu4e-utils hides ~/Repos/el/mu/mu4e/mu4e-utils
~/Repos/el/mu/build/mu4e/mu4e-context hides ~/Repos/el/mu/mu4e/mu4e-context
~/Repos/el/mu/build/mu4e/mu4e-draft hides ~/Repos/el/mu/mu4e/mu4e-draft
~/Repos/el/mu/build/mu4e/mu4e-message hides ~/Repos/el/mu/mu4e/mu4e-message
~/Repos/el/mu/build/mu4e/mu4e-compose hides ~/Repos/el/mu/mu4e/mu4e-compose
~/Repos/el/mu/build/mu4e/mu4e-view-common hides ~/Repos/el/mu/mu4e/mu4e-view-common
~/Repos/el/mu/build/mu4e/mu4e-view-old hides ~/Repos/el/mu/mu4e/mu4e-view-old
~/Repos/el/mu/build/mu4e/mu4e-view-gnus hides ~/Repos/el/mu/mu4e/mu4e-view-gnus
~/Repos/el/mu/build/mu4e/mu4e-headers hides ~/Repos/el/mu/mu4e/mu4e-headers
~/Repos/el/mu/build/mu4e/mu4e-mark hides ~/Repos/el/mu/mu4e/mu4e-mark
~/Repos/el/mu/build/mu4e/mu4e-icalendar hides ~/Repos/el/mu/mu4e/mu4e-icalendar
~/Repos/el/mu/build/mu4e/mu4e-speedbar hides ~/Repos/el/mu/mu4e/mu4e-speedbar
~/Repos/el/mu/build/mu4e/mu4e-contrib hides ~/Repos/el/mu/mu4e/mu4e-contrib
~/Repos/el/mu/build/mu4e/mu4e-proc hides ~/Repos/el/mu/mu4e/mu4e-proc
~/Repos/el/mu/build/mu4e/mu4e-meta hides ~/Repos/el/mu/mu4e/mu4e-meta
~/Repos/el/mu/build/mu4e/mu4e-vars hides ~/Repos/el/mu/mu4e/mu4e-vars
/home/horn/.emacs.d/elpa/transient-20210530.2252/transient hides /home/horn/Repos/el/emacs/lisp/transient

Features:
(shadow view bs emacsbug dired-aux company-oddmuse company-keywords
company-etags etags fileloop generator xref project company-gtags
company-dabbrev-code company-dabbrev company-files company-clang
company-capf company-cmake company-semantic company-template
company-bbdb paredit vc-mtn vc-hg vc-bzr vc-src vc-sccs vc-svn vc-cvs
vc-rcs debbugs-browse display-fill-column-indicator auto-package-update
finder-inf generic yaml-mode fish-mode cargo cargo-process rust-utils
rust-mode rust-rustfmt rust-playpen rust-compile compile rust-cargo
web-mode disp-table preview-latex auto-loads tex-site hl-todo
aggressive-indent rainbow-mode vc-git vc-dir ewoc vc vc-dispatcher
epa-file diredfl dired-x mu4e-alert time ht s mu4e-icalendar
gnus-icalendar org-capture org-refile icalendar diary-lib diary-loaddefs
mu4e mu4e-org mu4e-main mu4e-view mu4e-view-gnus mu4e-view-common
mu4e-headers mu4e-compose mu4e-context mu4e-draft mu4e-actions ido
rfc2368 mu4e-mark mu4e-proc mu4e-utils doc-view jka-compr image-mode
exif mu4e-lists mu4e-message flow-fill org ob ob-tangle ob-ref ob-lob
ob-table ob-exp org-macro org-footnote org-src ob-comint org-pcomplete
org-list org-faces org-entities org-version ob-emacs-lisp ob-core
ob-eval org-table ol org-keys org-compat org-macs org-loaddefs find-func
cal-menu calendar cal-loaddefs mule-util hl-line mu4e-vars mu4e-meta
smtpmail-multi smtpmail sendmail ecomplete completion auto-dictionary
flyspell ispell tramp-smb which-key highlight-parentheses
company-restclient know-your-http-well http-status-codes http-relations
http-methods http-headers restclient company-posframe posframe company
pcase yasnippet forge-list forge-commands forge-semi forge-bitbucket
buck forge-gogs gogs forge-gitea gtea forge-gitlab glab forge-github
ghub-graphql treepy gsexp ghub let-alist gnutls forge-notify
forge-revnote forge-pullreq forge-issue forge-topic forge-post
markdown-mode color thingatpt noutline outline forge-repo forge
forge-core forge-db closql emacsql-sqlite advice emacsql
emacsql-compiler magit-submodule magit-obsolete magit-blame magit-stash
magit-reflog magit-bisect magit-push magit-pull magit-fetch magit-clone
magit-remote magit-commit magit-sequence magit-notes magit-worktree
magit-tag magit-merge magit-branch magit-reset magit-files magit-refs
magit-status magit magit-repos magit-apply magit-wip magit-log
which-func imenu magit-diff smerge-mode diff diff-mode git-commit
log-edit pcvs-util add-log magit-core magit-autorevert autorevert
filenotify magit-margin magit-transient magit-process with-editor server
magit-mode transient magit-git magit-section magit-utils crm dash
visual-filename-abbrev debbugs soap-client url-http url-auth url-gw nsm
warnings rng-xsd rng-dt rng-util xsd-regexp bug-reference vertico
marginalia use-package-bind-key bind-key easy-mmode icomplete mb-depth
use-package-diminish ace-window avy alert log4e notifications gntp rx
tramp-cache tramp-sh recentf tree-widget pixel-scroll saveplace savehist
paren smiley gnus-art mm-uu mml2015 mm-view mml-smime smime dig gnus-sum
shr kinsoku svg dom gnus-group gnus-undo gnus-start gnus-dbus dbus xml
gnus-cloud nnimap nnmail mail-source utf7 netrc nnoo gnus-spec gnus-int
gnus-range message rmc puny dired dired-loaddefs rfc822 mml mml-sec epa
derived epg epg-config mm-decode mm-bodies mm-encode mail-parse rfc2231
mailabbrev gmm-utils mailheader gnus-win gnus wid-edit nnheader
gnus-util rmail rmail-loaddefs rfc2047 rfc2045 ietf-drums
text-property-search mm-util mail-prsvr mail-utils edmacro kmacro
doom-themes-ext-org doom-themes-ext-visual-bell face-remap
doom-vibrant-theme doom-themes doom-themes-base diminish cl-extra
help-mode use-package-ensure use-package-core tramp tramp-loaddefs
trampver tramp-integration files-x tramp-compat shell pcomplete comint
ansi-color ring parse-time iso8601 time-date ls-lisp format-spec info
package browse-url url url-proxy url-privacy url-expand url-methods
url-history url-cookie url-domsuf url-util mailcap url-handlers
url-parse auth-source cl-seq eieio eieio-core cl-macs eieio-loaddefs
password-cache json subr-x map url-vars seq byte-opt gv bytecomp
byte-compile cconv cl-loaddefs cl-lib iso-transl tooltip eldoc electric
uniquify ediff-hook vc-hooks lisp-float-type mwheel term/x-win x-win
term/common-win x-dnd tool-bar dnd fontset image regexp-opt fringe
tabulated-list replace newcomment text-mode elisp-mode lisp-mode
prog-mode register page tab-bar menu-bar rfn-eshadow isearch easymenu
timer select scroll-bar mouse jit-lock font-lock syntax font-core
term/tty-colors frame minibuffer cl-generic cham georgian utf-8-lang
misc-lang vietnamese tibetan thai tai-viet lao korean japanese eucjp-ms
cp51932 hebrew greek romanian slovak czech european ethiopic indian
cyrillic chinese composite charscript charprop case-table epa-hook
jka-cmpr-hook help simple abbrev obarray cl-preloaded nadvice button
loaddefs faces cus-face macroexp files window text-properties overlay
sha1 md5 base64 format env code-pages mule custom widget
hashtable-print-readable backquote threads dbusbind inotify lcms2
dynamic-setting system-font-setting font-render-setting cairo
move-toolbar gtk x-toolkit x multi-tty make-network-process emacs)

Memory information:
((conses 16 578262 70996)
 (symbols 48 44260 10)
 (strings 32 199185 6686)
 (string-bytes 1 6028236)
 (vectors 16 77469)
 (vector-slots 8 928820 30025)
 (floats 8 689 451)
 (intervals 56 1123 364)
 (buffers 992 16))